DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

How does AI transform chaos engineering from an experiment into a critical capability? Learn how to effectively operationalize the chaos.

Data quality isn't just a technical issue: It impacts an organization's compliance, operational efficiency, and customer satisfaction.

Are you a front-end or full-stack developer frustrated by front-end distractions? Learn to move forward with tooling and clear boundaries.

Developer Experience: Demand to support engineering teams has risen, and there is a shift from traditional DevOps to workflow improvements.

Coding

Also known as the build stage of the SDLC, coding focuses on the writing and programming of a system. The Zones in this category take a hands-on approach to equip developers with the knowledge about frameworks, tools, and languages that they can tailor to their own build needs.

Functions of Coding

Frameworks

Frameworks

A framework is a collection of code that is leveraged in the development process by providing ready-made components. Through the use of frameworks, architectural patterns and structures are created, which help speed up the development process. This Zone contains helpful resources for developers to learn about and further explore popular frameworks such as the Spring framework, Drupal, Angular, Eclipse, and more.

Java

Java

Java is an object-oriented programming language that allows engineers to produce software for multiple platforms. Our resources in this Zone are designed to help engineers with Java program development, Java SDKs, compilers, interpreters, documentation generators, and other tools used to produce a complete application.

JavaScript

JavaScript

JavaScript (JS) is an object-oriented programming language that allows engineers to produce and implement complex features within web browsers. JavaScript is popular because of its versatility and is preferred as the primary choice unless a specific function is needed. In this Zone, we provide resources that cover popular JS frameworks, server applications, supported data types, and other useful topics for a front-end engineer.

Languages

Languages

Programming languages allow us to communicate with computers, and they operate like sets of instructions. There are numerous types of languages, including procedural, functional, object-oriented, and more. Whether you’re looking to learn a new language or trying to find some tips or tricks, the resources in the Languages Zone will give you all the information you need and more.

Tools

Tools

Development and programming tools are used to build frameworks, and they can be used for creating, debugging, and maintaining programs — and much more. The resources in this Zone cover topics such as compilers, database management systems, code editors, and other software tools and can help ensure engineers are writing clean code.

Latest Premium Content
Trend Report
Developer Experience
Developer Experience
Trend Report
Low-Code Development
Low-Code Development
Refcard #400
Java Application Containerization and Deployment
Java Application Containerization and Deployment
Refcard #254
Apache Kafka Essentials
Apache Kafka Essentials

DZone's Featured Coding Resources

Create POM With LLM (GitHub Copilot) and Playwright MCP

Create POM With LLM (GitHub Copilot) and Playwright MCP

By Kailash Pathak DZone Core CORE
Test automation is a critical part of modern software development, but maintaining test scripts for dynamic web applications can be a challenge. The Page Object Model (POM) is a proven design pattern that makes test suites maintainable and scalable. When paired with GitHub Copilot, an AI-powered coding assistant, and Playwright’s Model Context Protocol (MCP), you can supercharge your automation workflow with intelligent code generation and seamless tool integration. In this blog, we’ll walk through how to create a POM-based test automation framework using Playwright, leverage GitHub Copilot to write and optimize code, and integrate Playwright MCP to enable AI-driven browser interactions. Whether you’re a QA engineer or a developer, this guide will help you build a robust, AI-enhanced testing solution. Page Object Model (POM) The Page Object Model is a design pattern that organizes test automation code by representing each web page or component as a class (a Page Object). These classes encapsulate the page’s elements (e.g., buttons, inputs) and interactions (e.g., clicking, typing), keeping test logic separate from UI manipulation. Benefits of POM Maintainability: Update one Page Object class when the UI changes, instead of rewriting multiple tests.Reusability: Reuse Page Objects across test cases to reduce code duplication.Readability: Write clear, business-focused test scripts that are easy to understand.Scalability: Modular code structure supports large, complex projects. GitHub Copilot: Your AI Coding Partner GitHub Copilot, powered by OpenAI’s Codex, is an AI-driven coding assistant integrated into IDEs like Visual Studio Code. It suggests code snippets, completes functions, and even generates entire classes based on your context or comments. For test automation, Copilot can: Generate boilerplate POM classes and test scripts.Suggest Playwright locators based on page descriptions.Write assertions and error-handling logic.Optimize existing code for better performance or readability. Playwright MCP: Bridging AI and Automation The Model Context Protocol (MCP) is an emerging standard (popularized by Anthropic and adopted by tools like Playwright) that enables AI models to interact with external systems, such as browsers, APIs, or databases. Think of MCP as a universal adapter that lets AI tools like Copilot control Playwright’s browser automation capabilities. With Playwright MCP, you can: Automate browser actions: Navigate pages, click elements, or fill forms via AI-driven commands.Integrate with AI: Allow Copilot to dynamically generate and execute Playwright commands.Integrate with database: Combine browser automation with API calls or database queries. Why Use GitHub Copilot and Playwright MCP Together? Using GitHub Copilot and Playwright MCP (Model Context Protocol) together enhances the development and testing workflow by combining AI-driven code assistance with advanced browser automation capabilities. Here’s why they are powerful when used together: Faster test creation: GitHub Copilot generates Playwright test scripts from natural language prompts, saving coding time.Reliable automation: Playwright MCP uses the accessibility tree for robust, cross-browser test execution.Enhanced productivity: Copilot suggests optimized code, while MCP automates browser tasks, streamlining workflows. Step-by-Step Guide: Building a POM With GitHub Copilot and Playwright MCP Here are the steps to set up Playwright Model Context Protocol (MCP) in Visual Studio Code for browser automation with GitHub Copilot: Step 1: Install Prerequisites Ensure Node.js (version 14 or higher) is installed. Verify with node -v and npm -v in a terminal. Download from nodejs.org if needed.Install Visual Studio Code (version 1.99 or later). Download from code.visualstudio.com.Install the GitHub Copilot extension in VS Code via the Extensions Marketplace. Step 2: Configure Playwright MCP Server In VS Code, open or create the settings.json file (File > Preferences > Settings > Open Settings (JSON)).Add the following configuration to enable the Playwright MCP server: JSON { "mcp": { "servers": { "playwright": { "command": "npx", "args": ["@playwright/mcp@latest"] } } }, } Step 3: Alternatively, Use the Command Shell code --add-mcp '{"name":"playwright","command":"npx","args":["@playwright/mcp@latest"]}' Step 4: Select Agent in GitHub Copilot Once all the above setups are done in GitHub Copilot, select “Agent.” Step 5: Verify the Installed Tool To verify whether the tool is installed properly, click on the tool icon and verify all the available tools, e.g., browser_close, browser_resize, etc. Create POM With GitHub Copilot and Playwright MCP Once all the above setup is complete, the next step is to provide instructions or prompts to the LLM (GitHub Copilot). Use Case For demo purposes, we are using the scenario below and asking GitHub Copilot and Playwright MCP to create a POM. Plain Text Create a POM model with the steps below: 1.Open https://www.saucedemo.com/ 2.Login with username and password 3.Add product “Sauce Labs Backpack” into the cart 4.Open the cart 5.Click on Checkout button 6.Fill random data in First Name,Last Name and Zip 7.Click on continue button 8.Click on Finish button 9.Verify message “Thank you for your order” Steps 1. Open the terminal. 2. Create a directory, e.g., mkdir MCP_DEMO. 3. Open the created directory in VS Code. 4. Now give the above use case step to GitHub Copilot. In the screenshot below, you can see how we can create a POM for the provided sites/steps in a few minutes. In the screenshot below, we can see the pages and test classes created in the respective folder. Video Conclusion GitHub Copilot and Playwright MCP help build robust automation frameworks in significantly less time. This AI-powered setup boosts productivity by accelerating code generation and simplifying browser interactions. However, while it streamlines development, the end user must always review and validate the generated code to ensure quality and accuracy. Despite this, the combination is a game-changer for QA engineers and developers aiming for scalable, maintainable, and intelligent test automation. More
Exploring the IBM App Connect Enterprise SELECT, ROW and THE Functions in ESQL

Exploring the IBM App Connect Enterprise SELECT, ROW and THE Functions in ESQL

By Matthias Blomme
Let’s talk about SELECT in ESQL—and I don’t mean database queries (although they are somewhat similar). I mean using SELECT to slice and dice message data inside IBM App Connect Enterprise (ACE). Think of it like SQL, but for navigating the message tree instead of a table. This post is all about the different ways you can use SELECT in ACE: returning arrays, rows, or just that one perfect value. To clarify, I’ve built out a quick demo flow with a very simple message structure, grabbed the outputs, and broken it all down for you. We’ll look at: Using plain SELECT to return a list of array items.Wrapping SELECT with ROW to structure the reply as a single array.Using THE to pull just one value. Here’s the input JSON I’m working with: JSON { "contact": { "info": { "name": "John", "lastName": "Wick", "title": "baba yaga" }, "details": { "phone": [{ "type": "business", "number": "911" }], "email": [ { "type": "personal", "address": "[email protected]" }, { "type": "business", "address": "[email protected]" }, { "type": "personal", "address": "[email protected]" } ] } } } All the source code and resources used in this blog can be downloaded from my GitHub repository. Plain SELECT – Returning a Message Tree Let’s start simple. We want to grab all the personal email addresses and return them as a JSON array. I’ll be using a very simple flow (the same set up for all three examples): Parsing the input data is done by the following ESQL code from the RetrieveEmailList Compute Node: The code block itself: JSON -- create proper json structure CREATE FIELD OutputRoot.JSON.Data.emailList IDENTITY(JSON.Array); -- select required data SET OutputRoot.JSON.Data.emailList.Item[] = ( SELECT U.address FROM InputRoot.JSON.Data.contact.details.email.Item[] AS U WHERE U.type = 'personal' ); And here's what we get back:{ "emailList": [ { "address": "[email protected]" }, { "address": "[email protected]" } ]}Note: That CREATE FIELD with IDENTITY(JSON.Array) is key — without it, you might run into JSON formatting weirdness or errors. Always prep your output structure when working with arrays in JSON. If you remove the JSON Array creation block, your output will look like this, { "emailList" : { "Item" : { "address" : "[email protected]" } } } Since the fields are not known to be an array, they overwrite each other on output, and you only get to see the last value. SELECT + ROW – Returning a Tree Structure Now let’s return the same data, but as a ROW. This can be useful when working with a single structured block of data, not loose array elements. The test flow: The ESQL code from the RetrieveEmailRow Compute Node: The code block: JSON -- create proper json structure CREATE FIELD OutputRoot.JSON.Data.emailRow IDENTITY(JSON.Array); -- select required data into a row SET OutputRoot.JSON.Data.emailRow = ROW ( SELECT U.address FROM InputRoot.JSON.Data.contact.details.email.Item[] AS U WHERE U.type = 'personal' ); Output: { "emailRow": [ { "address": "[email protected]" }, { "address": "[email protected]" } ]} Functionally this is still returning multiple items, but we’ve wrapped them in a ROW constructor. It’s subtly different in how ACE handles the result internally, especially if you plan to reuse or reference the result as a single variable downstream. Even though the resulting JSON looks the same, with or without the ROW cast, there is a difference in the internal structure. Internally, using ROW(...) changes how the data is handled in memory—it's treated as a single row structure rather than a collection of elements. This can be important when you’re manipulating the result later or passing it between compute nodes. Below you can see the structure without the ROW cast on the left, with the ROW cast on the right. For the data on the left, you must create an array, emailList, and select the data directly into the Item subfields, as you can see from the previous chapter. For the data on the right, the ROW function fills up the emailRow array. SELECT + THE – Grab One Specific Value This is where it gets fun. Let’s say we only want one business email—just the first match. This is where THE comes in handy. It returns a single scalar value from the result set—no array, no row, just the value. The flow I’ll be using: The RetrieveEmail ESQL code: The code block: JSON -- select the first match SET Environment.Variables.businessEmail = THE ( SELECT U.address FROM InputRoot.JSON.Data.contact.details.email.Item[] AS U WHERE U.type = 'business' ); -- return the value of the selected data SET OutputRoot.JSON.Data.businessEmail = Environment.Variables.businessEmail.address; Result: { "businessEmail": "[email protected]" } Note that I assign Environment.Variables.businessEmail.address to the OutputRoot and not Environment.Variables.businessEmail. THE gives you the first matching row, but when assigned to a variable, the structure is nested. That’s why we explicitly extract the address to flatten the response. If I did the same but changed the SELECT to retrieve personal emails, not business emails, I get the following result: Which is the first personal email in the returned list. I didn’t update the variable name but you get the idea. THE is your go-to when you're expecting only one value. If there are multiple matches, it'll return the first. If there are none—well, we all know what ESQL does with NULL values. SELECT THE ROW – Recap Technique Use Case Output Structure SELECT Retrieve a list of values JSON array elements ROW(SELECT…) Wrap the multiple values as a single row JSON array (internally treated as a row) THE(SELECT…) Return a single value (first match) Single value Beyond This Demo: SELECT Can Do More While this post focused on using SELECT to extract and shape data within a message, it’s worth noting that SELECT in ESQL is far more powerful. You can use it to: Transform message content into different structures.Join elements across multiple parts of a message tree.Translate values based on conditions or lookups.And even combine it with other ESQL features for more complex flows. It’s one of the most flexible and expressive tools in the ESQL toolbox—and this post is just a slice of what it can do. If you want to explore further, check out the links below. What About Java? Not a fan of ESQL? No worries, you can do the same thing using a JavaCompute Node. The syntax is different, but the logic is the same: extract values, structure your output, and go. Below are two ways to approach it: manual traversal and XPath (for XML only). Let’s say we want to do the following: Get all personal email addresses (similar to our SELECT ... WHERE U.type = 'personal').Get the first business email (like THE(SELECT ...)). To do this in Java, you can use XPath or manually walk the MbElement tree. There are multiple ways of walking the MbElement tree in Java, we will just stick with one for now (more are coming in the following blog). Working With JSON (Manual Tree Traversal) Extracting all personal emails while walking the MbElement tree: If you prefer to copy-paste the user code block: JSON // Add user code below // Create OutputRoot JSON structure MbElement outRoot = outMessage.getRootElement(); MbElement jsonRoot = outRoot.createElementAsLastChild(MbJSON.PARSER_NAME); MbElement data = jsonRoot.createElementAsLastChild(MbElement.TYPE_NAME, MbJSON.DATA_ELEMENT_NAME, null); // Get the input email array MbElement emailArrayInput = inMessage.getRootElement() .getFirstElementByPath("/JSON/Data/contact/details/email"); // Create output emailArray array MbElement emailArray = data.createElementAsLastChild(MbJSON.ARRAY, "emailArray", null); // Init tracking for first business email MbElement inputRootElm = inMessage.getRootElement(); String businessEmail = null; // Loop through email elements once MbElement current = emailArrayInput.getFirstChild(); while (current != null) { MbElement typeElem = current.getFirstElementByPath("type"); MbElement addressElem = current.getFirstElementByPath("address"); if (typeElem != null && addressElem != null) { String type = typeElem.getValueAsString(); String address = addressElem.getValueAsString(); if ("personal".equals(type)) { MbElement item = emailArray.createElementAsLastChild(MbElement.TYPE_NAME, "Item", null); item.createElementAsLastChild(MbElement.TYPE_NAME_VALUE, "address", address); } if ("business".equals(type) && businessEmail == null) { businessEmail = address; } } current = current.getNextSibling(); } // Add businessEmail to output if found if (businessEmail != null) { data.createElementAsLastChild(MbElement.TYPE_NAME_VALUE, "businessEmail", businessEmail); } // End of user code Note: you don’t need to define type and address as separate string variables, it just makes debugging easier. Extracting the first business email (single value) with XPath is unfortunately only possible with XML messages and doesn’t work for JSON. That is why in the above example I’ve added the business email code inside the loop. The above code results in the following message; Working With XML (XPath) In case you are handling XML messages, you can extract the first business email using the following XPath expression: MbXPath xp = newMbXPath("/contact/details/email[type='business'][1]/address"); List<MbElement> nodeset = (List<MbElement>)inMessage.evaluateXPath(xp); Working that into a JavaCompute node gives you the following code: We’ll be using the following XML input message: XML <?xml version="1.0" encoding="UTF-8"?> <contact> <info> <name>John</name> <lastName>Wick</lastName> <title>baba yaga</title> </info> <details> <phone> <type>business</type> <number>911</number> </phone> <email> <type>personal</type> <address>[email protected]</address> </email> <email> <type>business</type> <address>[email protected]</address> </email> <email> <type>personal</type> <address>[email protected]</address> </email> </details> </contact> Which results in the following output message: Conclusion Whether you prefer ESQL or Java, the key takeaway is that ACE gives you flexible tools to extract and shape your data — you just need to know which approach fits your use case. We've looked at three special ESQL functions, using SELECT, ROW(...), and THE(...), and saw how the same logic can be implemented in Java using tree traversal (for JSON) or XPath (for XML). Hopefully, this helped demystify the different ways to “select the row,” and helps you optimize your ACE integrations. More
Memory Leak Due to Uncleared ThreadLocal Variables
Memory Leak Due to Uncleared ThreadLocal Variables
By Ram Lakshmanan DZone Core CORE
When Incentives Sabotage Product Strategy
When Incentives Sabotage Product Strategy
By Stefan Wolpers DZone Core CORE
KubeVirt: Can VM Management With Kubernetes Work?
KubeVirt: Can VM Management With Kubernetes Work?
By Chris Ward DZone Core CORE
Beyond Java Streams: Exploring Alternative Functional Programming Approaches in Java
Beyond Java Streams: Exploring Alternative Functional Programming Approaches in Java

Few concepts in Java software development have changed how we approach writing code in Java than Java Streams. They provide a clean, declarative way to process collections and have thus become a staple in modern Java applications. However, for all their power, Streams present their own challenges, especially where flexibility, composability, and performance optimization are priorities. What if your programming needs more expressive functional paradigms? What if you are looking for laziness and safety beyond what Streams provide and want to explore functional composition at a lower level? In this article, we will be exploring other functional programming techniques you can use in Java that do not involve using the Streams API. Java Streams: Power and Constraints Java Streams are built on a simple premise—declaratively process collections of data using a pipeline of transformations. You can map, filter, reduce, and collect data with clean syntax. They eliminate boilerplate and allow chaining operations fluently. However, Streams fall short in some areas: They are not designed for complex error handling.They offer limited lazy evaluation capabilities.They don’t integrate well with asynchronous processing.They lack persistent and immutable data structures. One of our fellow DZone members wrote a very good article on "The Power and Limitations of Java Streams," which describes both the advantages and limitations of what you can do using Java Streams. I agree that Streams provide a solid basis for functional programming, but I suggest looking around for something even more powerful. The following alternatives are discussed within the remainder of this article, expanding upon points introduced in the referenced piece. Vavr: A Functional Java Library Why Vavr? Provides persistent and immutable collections (e.g., List, Set, Map)Includes Try, Either, and Option types for robust error handlingSupports advanced constructs like pattern matching and function composition Vavr is often referred to as a "Scala-like" library for Java. It brings in a strong functional flavor that bridges Java's verbosity and the expressive needs of functional paradigms. Example: Java Option<String> name = Option.of("Bodapati"); String result = name .map(n -> n.toUpperCase()) .getOrElse("Anonymous"); System.out.println(result); // Output: BODAPATI Using Try, developers can encapsulate exceptions functionally without writing try-catch blocks: Java Try<Integer> safeDivide = Try.of(() -> 10 / 0); System.out.println(safeDivide.getOrElse(-1)); // Output: -1 Vavr’s value becomes even more obvious in concurrent and microservice environments where immutability and predictability matter. Reactor and RxJava: Going Asynchronous Reactive programming frameworks such as Project Reactor and RxJava provide more sophisticated functional processing streams that go beyond what Java Streams can offer, especially in the context of asynchrony and event-driven systems. Key Features: Backpressure control and lazy evaluationAsynchronous stream compositionRich set of operators and lifecycle hooks Example: Java Flux<Integer> numbers = Flux.range(1, 5) .map(i -> i * 2) .filter(i -> i % 3 == 0); numbers.subscribe(System.out::println); Use cases include live data feeds, user interaction streams, and network-bound operations. In the Java ecosystem, Reactor is heavily used in Spring WebFlux, where non-blocking systems are built from the ground up. RxJava, on the other hand, has been widely adopted in Android development where UI responsiveness and multithreading are critical. Both libraries teach developers to think reactively, replacing imperative patterns with a declarative flow of data. Functional Composition with Java’s Function Interface Even without Streams or third-party libraries, Java offers the Function<T, R> interface that supports method chaining and composition. Example: Java Function<Integer, Integer> multiplyBy2 = x -> x * 2; Function<Integer, Integer> add10 = x -> x + 10; Function<Integer, Integer> combined = multiplyBy2.andThen(add10); System.out.println(combined.apply(5)); // Output: 20 This simple pattern is surprisingly powerful. For example, in validation or transformation pipelines, you can modularize each logic step, test them independently, and chain them without side effects. This promotes clean architecture and easier testing. JEP 406 — Pattern Matching for Switch Pattern matching, introduced in Java 17 as a preview feature, continues to evolve and simplify conditional logic. It allows type-safe extraction and handling of data. Example: Java static String formatter(Object obj) { return switch (obj) { case Integer i -> "Integer: " + i; case String s -> "String: " + s; default -> "Unknown type"; }; } Pattern matching isn’t just syntactic sugar. It introduces a safer, more readable approach to decision trees. It reduces the number of nested conditions, minimizes boilerplate, and enhances clarity when dealing with polymorphic data. Future versions of Java are expected to enhance this capability further with deconstruction patterns and sealed class integration, bringing Java closer to pattern-rich languages like Scala. Recursion and Tail Call Optimization Workarounds Recursion is fundamental in functional programming. However, Java doesn’t optimize tail calls, unlike languages like Haskell or Scala. That means recursive functions can easily overflow the stack. Vavr offers a workaround via trampolines: Java static Trampoline<Integer> factorial(int n, int acc) { return n == 0 ? Trampoline.done(acc) : Trampoline.more(() -> factorial(n - 1, n * acc)); } System.out.println(factorial(5, 1).result()); Trampolining ensures that recursive calls don’t consume additional stack frames. Though slightly verbose, this pattern enables functional recursion in Java safely. Conclusion: More Than Just Streams "The Power and Limitations of Java Streams" offers a good overview of what to expect from Streams, and I like how it starts with a discussion on efficiency and other constraints. So, I believe Java functional programming is more than just Streams. There is a need to adopt libraries like Vavr, frameworks like Reactor/RxJava, composition, pattern matching, and recursion techniques. To keep pace with the evolution of the Java enterprise platform, pursuing hybrid patterns of functional programming allows software architects to create systems that are more expressive, testable, and maintainable. Adopting these tools doesn’t require abandoning Java Streams—it means extending your toolbox. What’s Next? Interested in even more expressive power? Explore JVM-based functional-first languages like Kotlin or Scala. They offer stronger FP constructs, full TCO, and tighter integration with functional idioms. Want to build smarter, more testable, and concurrent-ready Java systems? Time to explore functional programming beyond Streams. The ecosystem is richer than ever—and evolving fast. What are your thoughts about functional programming in Java beyond Streams? Let’s talk in the comments!

By Rama Krishna Prasad Bodapati
How to Install and Set Up Jenkins With Docker Compose
How to Install and Set Up Jenkins With Docker Compose

Jenkins is an open-source CI/CD tool written in Java that is used for organising the CI/CD pipelines. Currently, at the time of writing this blog, it has 24k stars and 9.1k forks on GitHub. With over 2000 plugin support, Jenkins is a well-known tool in the DevOps world. The following are multiple ways to install and set up Jenkins: Using the Jenkins Installer package for WindowsUsing Homebrew for macOSUsing the Generic Java Package (war)Using DockerUsing KubernetesUsing apt for Ubuntu/Debian Linux OS In this tutorial blog, I will cover the step-by-step process to install and setup Jenkins using Docker Compose for an efficient and seamless CI/CD experience. Using Dockerwith Jenkins allows users to set up a Jenkins instance quickly with minimal manual configuration. It ensures portability and scalability, as with Docker Compose, users can easily set up Jenkins and its required services, such as volumes and networks, using a single YAML file. This allows the users to easily manage and replicate the setup in different environments. Installing Jenkins Using Docker Compose Installing Jenkins with Docker Compose makes the setup process simple and efficient, and allows us to define configurations in a single file. This approach removes the complexity and difficulty faced while installing Jenkins manually and ensures easy deployment, portability, and quick scaling. Prerequisite As a prerequisite, Docker Desktop needs to be installed, up and running on the local machine. Docker Compose is included in Docker Desktop along with Docker Engine and Docker CLI. Jenkins With Docker Compose Jenkins could be instantly set up by running the following docker-compose command using the terminal: Plain Text docker compose up -d This docker-compose command could be run by navigating to the folder where the Docker Compose file is placed. So, let’s create a new folder jenkins-demo and inside this folder, let’s create another new folder jenkins-configuration and a new file docker-compose.yaml. The following is the folder structure: Plain Text jenkins-demo/ ├── jenkins-configuration/ └── docker-compose.yaml The following content should be added to the docker-compose.yaml file. YAML # docker-compose.yaml version: '3.8' services: jenkins: image: jenkins/jenkins:lts privileged: true user: root ports: - 8080:8080 - 50000:50000 container_name: jenkins volumes: - /Users/faisalkhatri/jenkins-demo/jenkins-configuration:/var/jenkins_home - /var/run/docker.sock:/var/run/docker.sock Decoding the Docker Compose File The first line in the file is a comment. The services block starts from the second line, which includes the details of the Jenkins service. The Jenkins service block contains the image, user, and port details. The Jenkins service will run the latest Jenkins image with root privileges and name the container as jenkins. The ports are responsible for mapping container ports to the host machine. The details of these ports are as follows: 8080:8080:This will map the port 8080 inside the container to the port 8080 on the host machine. It is important, as it is required for accessing the Jenkins web interface. It will help us in accessing Jenkins in the browser by navigating to http://localhost:808050000:50000:This will map the port 50000 inside the container to port 50000 on the host machine. It is the JNLP (Java Network Launch Protocol) agent port, which is used for connecting Jenkins build agents to the Jenkins Controller instance. It is important, as we would be using distributed Jenkins setups, where remote build agents connect to the Jenkins Controller instance. The privileged: true setting will grant the container full access to the host system and allow running the process as the root user on the host machine. This will enable the container to perform the following actions : Access all the host devicesModify the system configurationsMount file systemsManage network interfacesPerform admin tasks that a regular container cannot perform These actions are important, as Jenkins may require permissions to run specific tasks while interacting with the host system, like managing Docker containers, executing system commands, or modifying files outside the container. Any data stored inside the container is lost when the container stops or is removed. To overcome this issue, Volumes are used in Docker to persist data beyond the container’s lifecycle. We will use Docker Volumes to keep the Jenkins data intact, as it is needed every time we start Jenkins. Jenkins data would be stored in the jenkins-configuration folder on the local machine. The /Users/faisalkhatri/jenkins-demo/jenkins-configuration on the host is mapped to /var/jenkins_home in the container. The changes made inside the container in the respective folder will reflect on the folder on the host machine and vice versa. This line /var/run/docker.sock:/var/run/docker.sock, mounts the Docker socket from the host into the container, allowing the Jenkins container to directly communicate with the Docker daemon running on the host machine. This enables Jenkins, which is running inside the container, to manage and run Docker commands on the host, allowing it to build and run other Docker containers as a part of CI/CD pipelines. Installing Jenkins With Docker Compose Let’s run the installation process step by step as follows: Step 1 — Running Jenkins Setup Open a terminal, navigate to the jenkins-demo folder, and run the following command: Plain Text docker compose up -d After the command is successfully executed, open any browser on your machine and navigate to https://localhost:8080, you should be able to find the Unlock Jenkins screen as shown in the screenshot below: Step 2 — Finding the Jenkins Password From the Docker Container The password to unlock Jenkins could be found by navigating to the jenkins container (remember we had given the name jenkins to the container in the Docker Compose file) and checking out its logs by running the following command on the terminal: Plain Text docker logs jenkins Copy the password from the logs, paste it in the Administrator password field on the Unlock Jenkins screen in the browser, and click on the Continue button. Step 3 — Setting up Jenkins The “Getting Started” screen will be displayed next, which will prompt us to install plugins to set up Jenkins. Select the Install suggested plugins and proceed with the installation. It will take some time for the installations to complete. Step 4 — Creating Jenkins user After the installation is complete, Jenkins will show the next screen to update the user details. It is recommended to update the user details with a password and click on Save and Continue. This username and password can then be used to log in to Jenkins. Step 5 — Instance Configuration In this window, we can update the Jenkins accessible link so it can be further used to navigate and run Jenkins. However, we can leave it as it is now — http://localhost:8080. Click on the Save and Finish button to complete the set up. With this, the Jenkins installation and set up are complete; we are now ready to use Jenkins. Summary Docker is the go-to tool for instantly spinning up a Jenkins instance. Using Docker Compose, we installed Jenkins successfully in just 5 simple steps. Once Jenkins is up and started, we can install the required plugin and set up CI/CD workflows as required. Using Docker Volumes allows us to use Jenkins seamlessly, as it saves the instance data between restarts. In the next tutorial, we will learn about installing and setting up Jenkins agents that will help us run the Jenkins jobs.

By Faisal Khatri DZone Core CORE
Mastering Fluent Bit: Controlling Logs With Fluent Bit on Kubernetes (Part 4)
Mastering Fluent Bit: Controlling Logs With Fluent Bit on Kubernetes (Part 4)

This series is a general-purpose getting-started guide for those of us wanting to learn about the Cloud Native Computing Foundation (CNCF) project Fluent Bit. Each article in this series addresses a single topic by providing insights into what the topic is, why we are interested in exploring that topic, where to get started with the topic, and how to get hands-on with learning about the topic as it relates to the Fluent Bit project. The idea is that each article can stand on its own, but that they also lead down a path that slowly increases our abilities to implement solutions with Fluent Bit telemetry pipelines. Let's take a look at the topic of this article, using Fluent Bit to get control of logs on a Kubernetes cluster. In case you missed the previous article, I'm providing a short introduction to Fluent Bit before sharing how to use Fluent Bit telemetry pipeline on a Kubernetes cluster to take control of all the logs being generated. What Is Fluent Bit? Before diving into Fluent Bit, let's step back and look at the position of this project within the Fluent organization. If we look at the Fluent organization on GitHub, we find the Fluentd and Fluent Bit projects hosted there. The backstory is that the project began as a log parsing project, using Fluentd, which joined the CNCF in 2026 and achieved Graduated status in 2019. Once it became apparent that the world was heading towards cloud-native Kubernetes environments, the solution was not designed to meet the flexible and lightweight requirements that Kubernetes solutions demanded. Fluent Bit was born from the need to have a low-resource, high-throughput, and highly scalable log management solution for cloud native Kubernetes environments. The project was started within the Fluent organization as a sub-project in 2017, and the rest is now a 10-year history in the release of v4 last week. Fluent Bit has become so much more than a flexible and lightweight log pipeline solution, now able to process metrics and traces, and becoming a telemetry pipeline collection tool of choice for those looking to put control over their telemetry data right at the source where it's being collected. Let's get started with Fluent Bit and see what we can do for ourselves! Why Control Logs on a Kubernetes Cluster? When you dive into the cloud native world, this means you are deploying containers on Kubernetes. The complexities increase dramatically as your applications and microservices interact in this complex and dynamic infrastructure landscape. Deployments can auto-scale, pods spin up and are taken down as the need arises, and underlying all of this are the various Kubernetes controlling components. All of these things are generating telemetry data, and Fluent Bit is a wonderfully simple way to take control of them across a Kubernetes cluster. It provides a way of collecting everything through a central telemetry pipeline as you go, while providing the ability to parse, filter, and route all your telemetry data. For developers, this article will demonstrate using Fluent Bit as a single point of log collection on a development Kubernetes cluster with a deployed workload. Finally, all examples in this article have been done on OSX and are assuming the reader is able to convert the actions shown here to their own local machines Where to Get Started To ensure you are ready to start controlling your Kubernetes cluster logs, the rest of this article assumes you have completed the previous article. This ensures you are running a two-node Kubernetes cluster with a workload running in the form of Ghost CMS, and Fluent Bit is installed to collect all container logs. If you did not work through the previous article, I've provided a Logs Control Easy Install project repository that you can download, unzip, and run with one command to spin up the Kubernetes cluster with the above setup on your local machine. Using either path, once set up, you are able to see the logs from Fluent Bit containing everything generated on this running cluster. This would be the logs across three namespaces: kube-system, ghost, and logging. You can verify that they are up and running by browsing those namespaces, shown here on my local machine: Go $ kubectl --kubeconfig target/2nodeconfig.yaml get pods --namespace kube-system NAME READY STATUS RESTARTS AGE coredns-668d6bf9bc-jrvrx 1/1 Running 0 69m coredns-668d6bf9bc-wbqjk 1/1 Running 0 69m etcd-2node-control-plane 1/1 Running 0 69m kindnet-fmf8l 1/1 Running 0 69m kindnet-rhlp6 1/1 Running 0 69m kube-apiserver-2node-control-plane 1/1 Running 0 69m kube-controller-manager-2node-control-plane 1/1 Running 0 69m kube-proxy-b5vjr 1/1 Running 0 69m kube-proxy-jxpqc 1/1 Running 0 69m kube-scheduler-2node-control-plane 1/1 Running 0 69m $ kubectl --kubeconfig target/2nodeconfig.yaml get pods --namespace ghost NAME READY STATUS RESTARTS AGE ghost-dep-8d59966f4-87jsf 1/1 Running 0 77m ghost-dep-mysql-0 1/1 Running 0 77m $ kubectl --kubeconfig target/2nodeconfig.yaml get pods --namespace logging NAME READY STATUS RESTARTS AGE fluent-bit-7qjmx 1/1 Running 0 41m The initial configuration for the Fluent Bit instance is to collect all container logs, from all namespaces, shown in the fluent-bit-helm.yaml configuration file used in our setup, highlighted in bold below: Go args: - --workdir=/fluent-bit/etc - --config=/fluent-bit/etc/conf/fluent-bit.yaml config: extraFiles: fluent-bit.yaml: | service: flush: 1 log_level: info http_server: true http_listen: 0.0.0.0 http_port: 2020 pipeline: inputs: - name: tail tag: kube.* read_from_head: true path: /var/log/containers/*.log multiline.parser: docker, cri outputs: - name: stdout match: '*' To see all the logs collected, we can dump the Fluent Bit log file as follows, using the pod name we found above: Go $ kubectl --kubeconfig target/2nodeconfig.yaml logs fluent-bit-7qjmx --nanmespace logging [OUTPUT-CUT-DUE-TO-LOG-VOLUME] ... You will notice if you browse that you have error messages, info messages, if you look hard enough, some logs from Ghost's MySQL workload, the Ghost CMS workload, and even your Fluent Bit instance. As a developer working on your cluster, how can you find anything useful in this flood of logging? The good thing is you do have a single place to look for them! Another point to mention is that by using the Fluent Bit tail input plugin and setting it to read from the beginning of each log file, we have ensured that our log telemetry data is taken from all our logs. If we didn't set this to collect from the beginning of the log file, our telemetry pipeline would miss everything that was generated before the Fluent Bit instance started. This ensures we have the workload startup messages and can test on standard log telemetry events each time we modify our pipeline configuration. Let's start taking control of our logs and see how we, as developers, can make some use of the log data we want to see during our local development testing. Taking Back Control The first thing we can do is to focus our log collection efforts on just the workload we are interested in, and in this example, we are looking to find problems with our Ghost CMS deployment. As you are not interested in the logs from anything happening in the kube-system namespace, you can narrow the focus of your Fluent Bit input plugin to only examine Ghost log files. This can be done by making a new configuration file called myfluent-bit-heml.yaml file and changing the default path as follows in bold: Go args: - --workdir=/fluent-bit/etc - --config=/fluent-bit/etc/conf/fluent-bit.yaml config: extraFiles: fluent-bit.yaml: | service: flush: 1 log_level: info http_server: true http_listen: 0.0.0.0 http_port: 2020 pipeline: inputs: - name: tail tag: kube.* read_from_head: true path: /var/log/containers/*ghost* multiline.parser: docker, cri outputs: - name: stdout match: '*' The next step is to update the Fluent Bit instance with a helm update command as follows: Go $ helm upgrade --kubeconfig target/2nodeconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml NAME READY STATUS RESTARTS AGE fluent-bit-mzktk 1/1 Running 0 28s Now, explore the logs being collected by Fluent Bit and notice that all the kube-system namespace logs are no longer there, and we can focus on our deployed workload. Go $ kubectl --kubeconfig target/2nodeconfig.yaml logs fluent-bit-mzktk --nanmespace logging ... [11] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583486.278137067, {}], {"time"=>"2025-05-18T15:51:26.278137067Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:51:26.27 INFO ==> Configuring database"}] [12] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583486.318427288, {}], {"time"=>"2025-05-18T15:51:26.318427288Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:51:26.31 INFO ==> Setting up Ghost"}] [13] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583491.211337893, {}], {"time"=>"2025-05-18T15:51:31.211337893Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:51:31.21 INFO ==> Configuring Ghost URL to http://127.0.0.1:2368"}] [14] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583491.234609188, {}], {"time"=>"2025-05-18T15:51:31.234609188Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:51:31.23 INFO ==> Passing admin user creation wizard"}] [15] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583491.243222300, {}], {"time"=>"2025-05-18T15:51:31.2432223Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:51:31.24 INFO ==> Starting Ghost in background"}] [16] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583519.424206501, {}], {"time"=>"2025-05-18T15:51:59.424206501Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:51:59.42 INFO ==> Stopping Ghost"}] [17] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583520.921096963, {}], {"time"=>"2025-05-18T15:52:00.921096963Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:52:00.92 INFO ==> Persisting Ghost installation"}] [18] kube.var.log.containers.ghost-dep-8d59966f4-87jsf_ghost_ghost-dep-c8ee31893743a1ce781f6f43ea3d0bfb93412623a721a2248e842936dc567089.log: [[1747583521.008567054, {}], {"time"=>"2025-05-18T15:52:01.008567054Z", "stream"=>"stderr", "_p"=>"F", "log"=>"ghost 15:52:01.00 INFO ==> ** Ghost setup finished! **"}] ... This is just a selection of log lines from the total output. If you look closer, you see these logs have their own sort of format, so let's standardize them so that JSON is the output format and make the various timestamps a bit more readable by changing your Fluent Bit output plugin configuration as follows: Go args: - --workdir=/fluent-bit/etc - --config=/fluent-bit/etc/conf/fluent-bit.yaml config: extraFiles: fluent-bit.yaml: | service: flush: 1 log_level: info http_server: true http_listen: 0.0.0.0 http_port: 2020 pipeline: inputs: - name: tail tag: kube.* read_from_head: true path: /var/log/containers/*ghost* multiline.parser: docker, cri outputs: - name: stdout match: '*' format: json_lines json_date_format: java_sql_timestamp Update the Fluent Bit instance using a helm update command as follows: Go $ helm upgrade --kubeconfig target/2nodeconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml NAME READY STATUS RESTARTS AGE fluent-bit-gqsc8 1/1 Running 0 42s Now, explore the logs being collected by Fluent Bit and notice the output changes: Go $ kubectl --kubeconfig target/2nodeconfig.yaml logs fluent-bit-gqsc8 --nanmespace logging ... {"date":"2025-06-05 13:49:58.001603","time":"2025-06-05T13:49:58.001603337Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:58.00 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Stopping Ghost"} {"date":"2025-06-05 13:49:59.291618","time":"2025-06-05T13:49:59.291618721Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:59.29 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Persisting Ghost installation"} {"date":"2025-06-05 13:49:59.387701","time":"2025-06-05T13:49:59.38770119Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:59.38 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> ** Ghost setup finished! **"} {"date":"2025-06-05 13:49:59.387736","time":"2025-06-05T13:49:59.387736981Z","stream":"stdout","_p":"F","log":""} {"date":"2025-06-05 13:49:59.451176","time":"2025-06-05T13:49:59.451176821Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:59.45 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> ** Starting Ghost **"} {"date":"2025-06-05 13:50:00.171207","time":"2025-06-05T13:50:00.171207812Z","stream":"stdout","_p":"F","log":""} ... Now, if we look closer at the array of messages and being the developer we are, we've noticed a mix of stderr and stdout log lines. Let's take control and trim out all the lines that do not contain stderr, as we are only interested in what is broken. We need to add a filter section to our Fluent Bit configuration using the grep filter and targeting a regular expression to select the keys stream or stderr as follows: Go args: - --workdir=/fluent-bit/etc - --config=/fluent-bit/etc/conf/fluent-bit.yaml config: extraFiles: fluent-bit.yaml: | service: flush: 1 log_level: info http_server: true http_listen: 0.0.0.0 http_port: 2020 pipeline: inputs: - name: tail tag: kube.* read_from_head: true path: /var/log/containers/*ghost* multiline.parser: docker, cri filters: - name: grep match: '*' regex: stream stderr outputs: - name: stdout match: '*' format: json_lines json_date_format: java_sql_timestamp Update the Fluent Bit instance using a helm update command as follows: Go $ helm upgrade --kubeconfig target/2nodeconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml NAME READY STATUS RESTARTS AGE fluent-bit-npn8n 1/1 Running 0 12s Now, explore the logs being collected by Fluent Bit and notice the output changes: Go $ kubectl --kubeconfig target/2nodeconfig.yaml logs fluent-bit-npn8n --nanmespace logging ... {"date":"2025-06-05 13:49:34.807524","time":"2025-06-05T13:49:34.807524266Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:34.80 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Configuring database"} {"date":"2025-06-05 13:49:34.860722","time":"2025-06-05T13:49:34.860722188Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:34.86 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Setting up Ghost"} {"date":"2025-06-05 13:49:36.289847","time":"2025-06-05T13:49:36.289847086Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:36.28 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Configuring Ghost URL to http://127.0.0.1:2368"} {"date":"2025-06-05 13:49:36.373376","time":"2025-06-05T13:49:36.373376803Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:36.37 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Passing admin user creation wizard"} {"date":"2025-06-05 13:49:36.379461","time":"2025-06-05T13:49:36.379461971Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:36.37 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Starting Ghost in background"} {"date":"2025-06-05 13:49:58.001603","time":"2025-06-05T13:49:58.001603337Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:58.00 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Stopping Ghost"} {"date":"2025-06-05 13:49:59.291618","time":"2025-06-05T13:49:59.291618721Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:59.29 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Persisting Ghost installation"} {"date":"2025-06-05 13:49:59.387701","time":"2025-06-05T13:49:59.38770119Z","stream":"stderr","_p":"F","log":"\u001b[38;5;6mghost \u001b[38;5;5m13:49:59.38 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> ** Ghost setup finished! **"} ... We are no longer seeing standard output log events, as our telemetry pipeline is now filtering to only show standard error-tagged logs! This exercise has shown how to format and prune our logs using our Fluent Bit telemetry pipeline on a Kubernetes cluster. Now let's look at how to enrich our log telemetry data. We are going to add tags to every standard error line pointing the on-call developer to the SRE they need to contact. To do this, we expand our filter section of the Fluent Bit configuration using the modify filter and targeting the keys stream or stderr to remove those keys and add two new keys, STATUS and ACTION, as follows: Go args: - --workdir=/fluent-bit/etc - --config=/fluent-bit/etc/conf/fluent-bit.yaml config: extraFiles: fluent-bit.yaml: | service: flush: 1 log_level: info http_server: true http_listen: 0.0.0.0 http_port: 2020 pipeline: inputs: - name: tail tag: kube.* read_from_head: true path: /var/log/containers/*ghost* multiline.parser: docker, cri filters: - name: grep match: '*' regex: stream stderr - name: modify match: '*' condition: Key_Value_Equals stream stderr remove: stream add: - STATUS REALLY_BAD - ACTION CALL_SRE outputs: - name: stdout match: '*' format: json_lines json_date_format: java_sql_timestamp Update the Fluent Bit instance using a helm update command as follows: Go $ helm upgrade --kubeconfig target/2nodeconfig.yaml --install fluent-bit fluent/fluent-bit --set image.tag=4.0.0 --namespace=logging --create-namespace --values=myfluent-bit-helm.yaml NAME READY STATUS RESTARTS AGE fluent-bit-ftfs4 1/1 Running 0 32s Now, explore the logs being collected by Fluent Bit and notice the output changes where the stream key is missing and two new ones have been added at the end of each error log event: Go $ kubectl --kubeconfig target/2nodeconfig.yaml logs fluent-bit-ftfs4 --nanmespace logging ... [CUT-LINE-FOR-VIEWING] Configuring database"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} [CUT-LINE-FOR-VIEWING] Setting up Ghost"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} [CUT-LINE-FOR-VIEWING] Configuring Ghost URL to http://127.0.0.1:2368"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} [CUT-LINE-FOR-VIEWING] Passing admin user creation wizard"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} [CUT-LINE-FOR-VIEWING] Starting Ghost in background"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} [CUT-LINE-FOR-VIEWING] Stopping Ghost"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} [CUT-LINE-FOR-VIEWING] Persisting Ghost installation"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} [CUT-LINE-FOR-VIEWING] ** Ghost setup finished! **"},"STATUS":"REALLY_BAD","ACTION":"CALL_SRE"} ... Now we have a running Kubernetes cluster, with two nodes generating logs, a workload in the form of a Ghost CMS generating logs, and using a Fluent Bit telemetry pipeline to gather and take control of our log telemetry data. Initially, we found that gathering all log telemetry data was flooding too much information to be able to sift out the important events for our development needs. We then started taking control of our log telemetry data by narrowing our collection strategy, by filtering, and finally by enriching our telemetry data. More in the Series In this article, you learned how to use Fluent Bit on a Kubernetes cluster to take control of your telemetry data. This article is based on this online free workshop. There will be more in this series as you continue to learn how to configure, run, manage, and master the use of Fluent Bit in the wild. Next up, integrating Fluent Bit telemetry pipelines with OpenTelemetry.

By Eric D. Schabell DZone Core CORE
Automating Sentiment Analysis Using Snowflake Cortex
Automating Sentiment Analysis Using Snowflake Cortex

In this hands-on tutorial, you'll learn how to automate sentiment analysis and categorize customer feedback using Snowflake Cortex, all through a simple SQL query without needing to build heavy and complex machine learning algorithms. No MLOps is required. We'll work with sample data simulating real customer feedback comments about a fictional company, "DemoMart," and classify each customer feedback entry using Cortex's built-in function. We'll determine sentiment (positive, negative, neutral) and label the feedback into different categories. The goal is to: Load a sample dataset of customer feedback into a Snowflake table.Use the built-in LLM-powered classification (CLASSIFY_TEXT) to tag each entry with a sentiment and classify the feedback into a specific category. Automate this entire workflow to run weekly using Snowflake Task.Generate insights from the classified data. Prerequisites A Snowflake account with access to Snowflake CortexRole privileges to create tables, tasks, and proceduresBasic SQL knowledge Step 1: Create Sample Feedback Table We'll use a sample dataset of customer feedback that covers products, delivery, customer support, and other areas. Let's create a table in Snowflake to store this data. Here is the SQL for creating the required table to hold customer feedback. SQL CREATE OR REPLACE TABLE customer.csat.feedback ( feedback_id INT, feedback_ts DATE, feedback_text STRING ); Now, you can load the data into the table using Snowflake's Snowsight interface. The sample data "customer_feedback_demomart.csv" is available in the GitHub repo. You can download and use it. Step 2: Use Cortex to Classify Sentiment and Category Let's read and process each row from the feedback table. Here's the magic. This single query classifies each piece of feedback for both sentiment and category: SQL SELECT feedback_id, feedback_ts, feedback_text, SNOWFLAKE.CORTEX.CLASSIFY_TEXT(feedback_text, ['positive', 'negative', 'neutral']):label::STRING AS sentiment, SNOWFLAKE.CORTEX.CLASSIFY_TEXT( feedback_text, ['Product', 'Customer Service', 'Delivery', 'Price', 'User Experience', 'Feature Request'] ):label::STRING AS feedback_category FROM customer.csat.feedback LIMIT 10; I have used the CLASSIFY_TEXT Function available within Snowflake's cortex to derive the sentiment based on the feedback_text and further classify it into a specific category the feedback is associated with, such as 'Product', 'Customer Service', 'Delivery', and so on. P.S.: You can change the categories based on your business needs. Step 3: Store Classified Results Let's store the classified results in a separate table for further reporting and analysis purposes. For this, I have created a table with the name feedback_classified as shown below. SQL CREATE OR REPLACE TABLE customer.csat.feedback_classified ( feedback_id INT, feedback_ts DATE, feedback_text STRING, sentiment STRING, feedback_category STRING ); Initial Bulk Load Now, let's do an initial bulk classification for all existing data before moving on to the incremental processing of newly arriving data. SQL -- Initial Load INSERT INTO customer.csat.feedback_classified SELECT feedback_id, feedback_ts, feedback_text, SNOWFLAKE.CORTEX.CLASSIFY_TEXT(feedback_text, ['positive', 'negative', 'neutral']):label::STRING, SNOWFLAKE.CORTEX.CLASSIFY_TEXT( feedback_text, ['Product', 'Customer Service', 'Delivery', 'Price', 'User Experience', 'Feature Request'] ):label::STRING AS feedback_label, CURRENT_TIMESTAMP AS PROCESSED_TIMESTAMP FROM customer.csat.feedback; Once the initial load is completed successfully, let's build an SQL that fetches only incremental data based on the processed_ts column value. For the incremental load, we need fresh data with customer feedback. For that, let's insert ten new records into our raw table customer.csat.feedback SQL INSERT INTO customer.csat.feedback (feedback_id, feedback_ts, feedback_text) VALUES (5001, CURRENT_DATE, 'My DemoMart order was delivered to the wrong address again. Very disappointing.'), (5002, CURRENT_DATE, 'I love the new packaging DemoMart is using. So eco-friendly!'), (5003, CURRENT_DATE, 'The delivery speed was slower than promised. Hope this improves.'), (5004, CURRENT_DATE, 'The product quality is excellent, I’m genuinely impressed with DemoMart.'), (5005, CURRENT_DATE, 'Customer service helped me cancel and reorder with no issues.'), (5006, CURRENT_DATE, 'DemoMart’s website was down when I tried to place my order.'), (5007, CURRENT_DATE, 'Thanks DemoMart for the fast shipping and great support!'), (5008, CURRENT_DATE, 'Received a damaged item. This is the second time with DemoMart.'), (5009, CURRENT_DATE, 'DemoMart app is very user-friendly. Shopping is a breeze.'), (5010, CURRENT_DATE, 'The feature I wanted is missing. Hope DemoMart adds it soon.'); Step 4: Automate Incremental Data Processing With TASK Now that we have newly added (incremental) fresh data into our raw table, let's create a task to pick up only new data and classify it automatically. We will schedule this task to run every Sunday at midnight UTC. SQL --Creating task CREATE OR REPLACE TASK CUSTOMER.CSAT.FEEDBACK_CLASSIFIED WAREHOUSE = COMPUTE_WH SCHEDULE = 'USING CRON 0 0 * * 0 UTC' -- Run evey Sunday at midnight UTC AS INSERT INTO customer.csat.feedback_classified SELECT feedback_id, feedback_ts, feedback_text, SNOWFLAKE.CORTEX.CLASSIFY_TEXT(feedback_text, ['positive', 'negative', 'neutral']):label::STRING, SNOWFLAKE.CORTEX.CLASSIFY_TEXT( feedback_text, ['Product', 'Customer Service', 'Delivery', 'Price', 'User Experience', 'Feature Request'] ):label::STRING AS feedback_label, CURRENT_TIMESTAMP AS PROCESSED_TIMESTAMP FROM customer.csat.feedback WHERE feedback_ts > (SELECT COALESCE(MAX(PROCESSED_TIMESTAMP),'1900-01-01') FROM CUSTOMER.CSAT.FEEDBACK_CLASSIFIED ); This will automatically run every Sunday at midnight UTC, process any newly arrived customer feedback, and classify it. Step 5: Visualize Insights You can now build dashboards in Snowsight to see weekly trends using a simple query like this: SQL SELECT feedback_category, sentiment, COUNT(*) AS total FROM customer.csat.feedback_classified GROUP BY feedback_category, sentiment ORDER BY total DESC; Conclusion With just a few lines of SQL, you: Ingested raw feedback into a Snowflake table.Used Snowflake Cortex to classify customer feedback and derive sentiment and feedback categoriesAutomated the process to run weeklyBuilt insights into the classified feedback for business users/leadership team to act upon by category and sentiment This approach is ideal for support teams, product teams, and leadership, as it allows them to continuously monitor customer experience without building or maintaining ML infrastructure. GitHub I have created a GitHub page with all the code and sample data. You can access it freely. The whole dataset generator and SQL scripts are available on GitHub.

By Rajanikantarao Vellaturi
Enterprise-Grade Distributed JMeter Load Testing on Kubernetes: A Scalable, CI/CD-Driven DevOps Approach
Enterprise-Grade Distributed JMeter Load Testing on Kubernetes: A Scalable, CI/CD-Driven DevOps Approach

Application performance, scalability, and resilience are critical for ensuring a seamless user experience. Apache JMeter is a powerful open-source tool for load testing, but running it on a single machine limits scalability, automation, and distributed execution. This blog presents a Kubernetes-powered JMeter setup on Azure Kubernetes Service (AKS), which can also be deployed on other cloud platforms like AWS EKS and Google GKE, integrated with CI/CD pipelines in Azure DevOps. This approach enables dynamic scaling, automated test execution, real-time performance monitoring, and automated reporting and alerting. Key Benefits of JMeter on AKS Run large-scale distributed load tests efficientlyAuto-scale worker nodes dynamically based on trafficAutomate test execution and result storage with CI/CDMonitor performance in real-time using InfluxDB & GrafanaGenerate automated reports and notify teams via email This guide follows a Kubernetes-native approach, leveraging: ConfigMaps for configuration managementDeployments for master and worker nodesServices for inter-node communicationHorizontal Pod Autoscaler (HPA) for dynamic scaling While this guide uses Azure DevOps as an example, the same approach can be applied to other CI/CD tools like Jenkins, GitHub Actions, or any automation framework with minimal modifications. For CI/CD integration, the same setup can be adapted for Jenkins, GitHub Actions, or any other CI/CD tool. Additionally, this JMeter setup is multi-cloud compatible, meaning it can be deployed on AWS EKS, Google GKE, or any Kubernetes environment. To fully automate the JMeter load simulation process, we integrate it with CI/CD pipelines, ensuring tests can be triggered on every code change, scheduled runs, or manually, while also enabling automated reporting and alerting to notify stakeholders of test results. What This Guide Covers Service Connection Setup – Authenticate AKS using Azure Service Principal.CI Pipeline Setup – Validate JMeter test scripts upon code commits.CD Pipeline Setup – Deploy and execute JMeter tests in a scalable environment.Performance Monitoring – Using InfluxDB and Grafana for real-time observability.Automated Reporting & Alerts – Convert JTL reports into HTML, extract key metrics, and send email notifications.Best Practices – Managing secrets securely and optimizing resource usage. If your system fails under heavy traffic, it could mean revenue loss, poor user experience, or even security risks. Traditional performance testing tools work well for small-scale tests, but what if you need to simulate thousands of concurrent users across multiple locations? This is where Kubernetes-powered JMeter comes in! By deploying JMeter on Azure Kubernetes Service (AKS) and integrating it with CI/CD Pipelines, we can: Run large-scale distributed tests efficientlyScale worker nodes dynamically based on loadAutomate the entire process, from deployment to reporting and result analysis Key Challenges with Traditional JMeter Execution Limitations of Running JMeter on a Single Machine Resource bottlenecks – Can’t simulate real-world distributed loads.Manual execution – No automation or CI/CD integration.Scalability issues – Hard to scale up or down dynamically.Data management – Handling large test datasets is cumbersome. Challenge JMeter on Local Machine JMeter on AKS Scalability Limited by CPU/memory Auto-scales with HPA Automation Manual test execution CI/CD pipelines for automation Parallel Execution Hard to distribute Kubernetes distributes the load Observability No centralized monitoring Grafana + InfluxDB integration Cost Efficiency Wasted resources On-demand scaling By deploying JMeter on AKS, we eliminate bottlenecks and achieve scalability, automation, and observability. JMeter Architecture on AKS A distributed JMeter deployment consists of: JMeter Master Pod – Orchestrates test execution.JMeter Worker Pods (Slaves) – Generate the actual load.JMeter Service – Enables inter-pod communication.InfluxDB – Stores real-time performance metrics.Grafana – Visualizes test execution.Azure File Storage – Stores test logs and results.Horizontal Pod Autoscaler (HPA) – Adjusts worker count based on CPU utilization. Figure 1: JMeter Distributed Load Testing Architecture on Azure Kubernetes Service (AKS), showing how the Master node orchestrates tests, Worker Pods generate load, and InfluxDB/Grafana monitor performance. Adding Real-World Use Cases To make the blog more relatable, let’s add examples of industries that benefit from scalable performance testing. E-commerce & Retail: Load testing before Black Friday & holiday sales.Banking & FinTech: Ensuring secure, high-performance online banking.Streaming Platforms: Handling millions of concurrent video streams.Healthcare Apps: Load-testing telemedicine platforms during peak hours.Gaming & Metaverse: Performance testing multiplayer online games. Optimizing Costs When Running JMeter on AKS Running JMeter on Azure Kubernetes Service (AKS) is powerful, but without optimization, it can get expensive. Let’s add a section on cost-saving strategies: Use Spot Instances for Non-Critical TestsAuto-Scale JMeter Worker Nodes Based on LoadSchedule Tests During Non-Peak Hours to Save CostsMonitor and Delete Unused Resources After Test ExecutionOptimize Log Storage – Avoid Keeping Large Log Files on AKS Deploying JMeter on AKS Prerequisites Ensure you have: Azure subscription with AKS configured. kubectl and helm installed. JMeter Docker images for master and worker nodes. JMX test plans and CSV datasets for load execution. Azure Service Principal for CI/CD automation. Creating JMeter Docker Images Your setup requires different Dockerfiles for the JMeter Master and JMeter Worker (Slave) nodes. Dockerfile - JMeter Master Shell FROM ubuntu:latest RUN apt-get update && apt-get install -y openjdk-11-jdk wget unzip WORKDIR /jmeter RUN wget https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.5.tgz && \ tar -xzf apache-jmeter-5.5.tgz && rm apache-jmeter-5.5.tgz CMD ["/jmeter/apache-jmeter-5.5/bin/jmeter"] Dockerfile - JMeter Worker (Slave) Shell FROM ubuntu:latest RUN apt-get update && apt-get install -y openjdk-11-jdk wget unzip WORKDIR /jmeter RUN wget https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.5.tgz && \ tar -xzf apache-jmeter-5.5.tgz && rm apache-jmeter-5.5.tgz CMD ["/bin/bash"] Once built and pushed to Azure Container Registry, these images will be used in Kubernetes deployments. Deploying InfluxDB for Performance Monitoring To capture real-time test results, deploy InfluxDB, which stores metrics from JMeter. File: jmeter_influxdb_configmap.yaml YAML apiVersion: v1 kind: ConfigMap metadata: name: influxdb-config labels: app: influxdb-jmeter data: influxdb.conf: | [meta] dir = "/var/lib/influxdb/meta" [data] dir = "/var/lib/influxdb/data" engine = "tsm1" wal-dir = "/var/lib/influxdb/wal" [[graphite]] enabled = true bind-address = ":2003" database = "jmeter" File: jmeter_influxdb_deploy.yaml YAML apiVersion: apps/v1 kind: Deployment metadata: name: influxdb-jmeter labels: app: influxdb-jmeter spec: replicas: 1 selector: matchLabels: app: influxdb-jmeter template: metadata: labels: app: influxdb-jmeter spec: containers: - image: influxdb name: influxdb volumeMounts: - name: config-volume mountPath: /etc/influxdb ports: - containerPort: 8086 volumes: - name: config-volume configMap: name: influxdb-config File: jmeter_influxdb_svc.yaml YAML apiVersion: v1 kind: Service metadata: name: jmeter-influxdb labels: app: influxdb-jmeter spec: ports: - port: 8086 name: api targetPort: 8086 selector: app: influxdb-jmeter Deployment Command Shell kubectl apply -f jmeter_influxdb_configmap.yaml kubectl apply -f jmeter_influxdb_deploy.yaml kubectl apply -f jmeter_influxdb_svc.yaml Verify InfluxDB Shell kubectl get pods -n <namespace-name> | grep influxdb Deploying Jmeter Master and Worker Nodes with Autoscaling Creating ConfigMap for JMeter Master - A ConfigMap is used to configure the JMeter master node. File: jmeter_master_configmap.yaml YAML apiVersion: v1 kind: ConfigMap metadata: name: jmeter-load-test labels: app: jmeter data: load_test: | #!/bin/bash /jmeter/apache-jmeter-*/bin/jmeter -n -t $1 -Dserver.rmi.ssl.disable=true -R $(getent ahostsv4 jmeter-slaves-svc | awk '{print $1}' | paste -sd ",") This script: Runs JMeter in non-GUI mode (-n).Disables RMI SSL for inter-pod communication.Dynamically resolves JMeter slave IPs. Deploying JMeter Master Nodes File: jmeter_master_deploy.yaml YAML apiVersion: apps/v1 kind: Deployment metadata: name: jmeter-master labels: app: jmeter-master spec: replicas: 1 selector: matchLabels: app: jmeter-master template: metadata: labels: app: jmeter-master spec: serviceAccountName: <Service Account Name> containers: - name: jmeter-master image: <your-jmeter-master-image> imagePullPolicy: IfNotPresent command: [ "/bin/bash", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] volumeMounts: - name: loadtest mountPath: /jmeter/load_test subPath: "load_test" - name: azure mountPath: /mnt/azure/jmeterresults ports: - containerPort: 60000 volumes: - name: loadtest configMap: name: jmeter-load-test defaultMode: 0777 - name: azure azureFile: secretName: files-secret shareName: jmeterresults readOnly: false This ensures: ConfigMap-based test executionPersistent storage for test resultsThe master node is always available Deploying JMeter Worker Nodes File: jmeter_slaves_deploy.yaml YAML apiVersion: apps/v1 kind: Deployment metadata: name: jmeter-slaves labels: app: jmeter-slave spec: replicas: 2 # Initial count, will be auto-scaled selector: matchLabels: app: jmeter-slave template: metadata: labels: app: jmeter-slave spec: serviceAccountName: <Service Account Name> containers: - name: jmeter-slave image: <your-jmeter-worker-image> imagePullPolicy: IfNotPresent volumeMounts: - name: azure mountPath: /mnt/azure/jmeterresults ports: - containerPort: 1099 - containerPort: 50000 volumes: - name: azure azureFile: secretName: files-secret shareName: jmeterresults readOnly: false Worker pods dynamically join the JMeter master and execute tests. Creating JMeter Worker Service File: jmeter_slaves_svc.yaml YAML apiVersion: v1 kind: Service metadata: name: jmeter-slaves-svc labels: app: jmeter-slave spec: clusterIP: None # Headless service for inter-pod communication ports: - port: 1099 targetPort: 1099 - port: 50000 targetPort: 50000 selector: app: jmeter-slave This enables JMeter master to discover worker nodes dynamically. Enabling Auto-Scaling for JMeter Workers File: jmeter_hpa.yaml YAML apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: jmeter-slaves-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: jmeter-slaves minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 Deploying All Components command Run the following command to deploy all components: Shell kubectl apply -f jmeter_master_configmap.yaml kubectl apply -f jmeter_master_deploy.yaml kubectl apply -f jmeter_slaves_deploy.yaml kubectl apply -f jmeter_slaves_svc.yaml kubectl apply -f jmeter_hpa.yaml To verify deployment: Shell kubectl get all -n <namespace-name> kubectl get hpa -n <namespace-name> kubectl get cm -n <namespace-name> Adding More Depth to Monitoring & Observability Performance testing is not just about running the tests—it’s about analyzing the results effectively. Using InfluxDB for Test Data StorageCreating Grafana Dashboards to Visualize TrendsIntegrating Azure Monitor & Log Analytics for Deeper Insights Example: Grafana Metrics for JMeter Performance Metric Description Response Time Measures how fast the system responds Throughput Requests per second handled Error Rate Percentage of failed requests CPU & Memory Usage Tracks AKS node utilization Deploying Grafana for Visualizing Test Results Once InfluxDB is running, configure Grafana to visualize the data. File: dashboard.sh Shell #!/usr/bin/env bash working_dir=`pwd` tenant=`awk '{print $NF}' $working_dir/tenant_export` grafana_pod=`kubectl get po -n $tenant | grep jmeter-grafana | awk '{print $1}'` kubectl exec -ti -n $tenant $grafana_pod -- curl 'http://admin:[email protected]:3000/api/datasources' -X POST -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"name":"jmeterdb","type":"influxdb","url":"http://jmeter-influxdb:8086","access":"proxy","isDefault":true,"database":"jmeter","user":"admin","password":"admin"}' Run Dashboard Script Shell chmod +x dashboard.sh ./dashboard.sh Automating Cluster Cleanup Once tests are complete, automate cleanup to free up resources. File: jmeter_cluster_delete.sh Shell #!/usr/bin/env bash clustername=$1 tenant=<namespace-name> echo "Deleting ConfigMaps" kubectl delete -n $tenant configmap jmeter-${clustername}-load-test echo "Deleting Jmeter Slaves" kubectl delete deployment.apps/jmeter-${clustername}-slaves kubectl delete service/jmeter-${clustername}-slaves-svc echo "Deleting Jmeter Master" kubectl delete deployment.apps/jmeter-${clustername}-master kubectl get -n $tenant all Run Cleanup Shell chmod +x jmeter_cluster_delete.sh ./jmeter_cluster_delete.sh <clustername> Running JMeter Tests Run a JMeter load test by executing the following in the master pod: Shell kubectl exec -ti jmeter-master -- /jmeter/load_test /mnt/azure/testplans/test.jmx -Gusers=100 -Gramp=10 This runs the test with: 100 concurrent users10-second ramp-up period Monitor Performance in Grafana Open Grafana UI (http://<Grafana-IP>:3000).View real-time results under the JMeter Dashboard. Stopping the JMeter Test To stop an active test: Shell kubectl exec -ti jmeter-master -- /jmeter/apache-jmeter-5.5/bin/stoptest.sh Automating JMeter Load Testing Using CI/CD Pipeline in Azure DevOps Figure 2: The CI/CD pipeline in Azure DevOps for automating JMeter execution, validating scripts, deploying to AKS, and storing results in Azure Blob Storage. Prerequisites for CI/CD in Azure DevOps Before creating the pipelines, ensure: Service Connection for AKS is set up using Azure App Registration / Service Principal with permissions to interact with AKS.Azure DevOps Agent (Self-hosted or Microsoft-hosted) is available to run the pipeline.Variable Groups & Key Vault Integration are configured for secure secrets management. Setting up Service Connection for AKS Create a Service Principal in Azure:az ad sp create-for-rbac --name "aks-service-connection" --role Contributor --scopes /subscriptions/<subscription-id> Go to Azure DevOps → Project Settings → Service Connections.Add a new Kubernetes Service Connection and authenticate using the Service Principal. Verify access using:az aks get-credentials --resource-group <resource-group> --name <aks-cluster> Setting Up CI/CD Pipelines for JMeter in Azure DevOps We will create two pipelines: CI Pipeline (Continuous Integration): Triggers when a commit happens and validates JMeter scripts.CD Pipeline (Continuous Deployment): Deploys JMeter to AKS and executes tests. Implementing the CI Pipeline (Validate JMeter Test Scripts) The CI pipeline will: Validate JMeter test scripts (.jmx)Check syntax and correctness Created File: azure-pipelines-ci.yml YAML trigger: branches: include: - main pool: vmImage: 'ubuntu-latest' steps: - checkout: self - task: UsePythonVersion@0 inputs: versionSpec: '3.x' - script: | echo "Validating JMeter Test Scripts" jmeter -n -t test_plan.jmx -l test_log.jtl displayName: "Validate JMeter Test Plan" Pipeline Execution: Saves logs (test_log.jtl) for debugging.Ensures no syntax errors before running tests in the CD pipeline. Implementing the CD Pipeline (Deploy & Execute JMeter Tests on AKS) The CD pipeline: Pulls the validated JMeter scripts.Deploys JMeter to AKS.Scales up worker nodes dynamically.Executes JMeter tests in distributed mode.Generates test reports and stores them in Azure Storage. Create File: azure-pipelines-cd.yml YAML trigger: - main pool: name: 'Self-hosted-agent' # Or use 'ubuntu-latest' for Microsoft-hosted agents variables: - group: "jmeter-variable-group" # Fetch secrets from Azure DevOps Variable Group stages: - stage: Deploy_JMeter displayName: "Deploy JMeter on AKS" jobs: - job: Deploy steps: - checkout: self - task: AzureCLI@2 displayName: "Login to Azure and Set AKS Context" inputs: azureSubscription: "$(azureServiceConnection)" scriptType: bash scriptLocation: inlineScript inlineScript: | az aks get-credentials --resource-group $(aksResourceGroup) --name $(aksClusterName) kubectl config use-context $(aksClusterName) - script: | echo "Deploying JMeter Master and Worker Nodes" kubectl apply -f jmeter_master_deploy.yaml kubectl apply -f jmeter_slaves_deploy.yaml kubectl apply -f jmeter_influxdb_deploy.yaml displayName: "Deploy JMeter to AKS" - script: | echo "Scaling Worker Nodes for Load Test" kubectl scale deployment jmeter-slaves --replicas=5 displayName: "Scale JMeter Workers" - stage: Execute_Load_Test displayName: "Run JMeter Load Tests" dependsOn: Deploy_JMeter jobs: - job: RunTest steps: - script: | echo "Executing JMeter Test Plan" kubectl exec -ti jmeter-master -- /jmeter/load_test /mnt/azure/testplans/test.jmx -Gusers=100 -Gramp=10 displayName: "Run JMeter Load Test" - script: | echo "Fetching JMeter Test Results" kubectl cp jmeter-master:/mnt/azure/jmeterresults/results test-results displayName: "Copy Test Results" - task: PublishPipelineArtifact@1 inputs: targetPath: "test-results" artifact: "JMeterTestResults" publishLocation: "pipeline" displayName: "Publish JMeter Test Results" Understanding the CD Pipeline Breakdown Step 1: Deploy JMeter on AKS Uses AzureCLI@2 to authenticate and set AKS context.Deploys JMeter Master, Worker nodes, and InfluxDB using YAML files. Step 2: Scale Worker Nodes Dynamically Uses kubectl scale to scale JMeter Worker pods based on test load. Step 3: Execute JMeter Load Test Runs the test using:kubectl exec -ti jmeter-master -- /jmeter/load_test /mnt/azure/testplans/test.jmx -Gusers=100 -Gramp=10 This triggers distributed execution. Step 4: Fetch & Publish Results Copies test results from JMeter Master pod.Publish the results as an artifact in Azure DevOps. Managing Secrets & Variables Securely To prevent exposing credentials: Use Variable Groups to store AKS names, resource groups, and secrets.Azure Key Vault Integration for storing sensitive information. YAML variables: - group: "jmeter-variable-group" Or directly use: YAML - task: AzureKeyVault@1 inputs: azureSubscription: "$(azureServiceConnection)" KeyVaultName: "my-keyvault" SecretsFilter: "*" Security Considerations in CI/CD Pipelines When integrating JMeter tests in Azure DevOps CI/CD Pipelines, security should be a priority. Use Azure Key Vault for Storing Secrets YAML - task: AzureKeyVault@1 inputs: azureSubscription: "$(azureServiceConnection)" KeyVaultName: "my-keyvault" SecretsFilter: "*" Limit AKS Access Using RBAC PoliciesEncrypt Test Data and CredentialsMonitor Pipeline Activities with Azure Security Center Automating Test Cleanup After Execution To free up AKS resources, the pipeline should scale down workers' post-test. Modify azure-pipelines-cd.yml YAML - script: | echo "Scaling Down JMeter Workers" kubectl scale deployment jmeter-slaves --replicas=1 displayName: "Scale Down Workers After Test" Best Practices for JMeter on AKS and CI/CD in Azure DevOps 1. Optimizing Performance and Scaling Optimize Auto-Scaling – Use HPA (Horizontal Pod Autoscaler) to dynamically adjust JMeter worker nodes.Optimize Worker Pods – Assign proper CPU and memory limits to avoid resource exhaustion.Store Results in Azure Storage – Prevent overload by saving JMeter logs in Azure Blob Storage.Automate Cleanup – Scale down JMeter workers post-test to save costs. Figure 3: Auto-Scaling of JMeter Worker Nodes using Horizontal Pod Autoscaler (HPA) in Azure Kubernetes Service (AKS), dynamically adjusting pod count based on CPU usage. 2. Monitoring and Observability Monitor Performance – Use InfluxDB + Grafana for real-time analysis.Use Azure Monitor & Log Analytics – Track AKS cluster health and performance.Integrate Grafana & Prometheus – (Optional) Provides visualization for live metrics.Automate Grafana Setup – Ensure seamless test monitoring and reporting.JMeter Logs & Metrics Collection – View live test logs using: kubectl logs -f jmeter-master 3. Best Practices for CI/CD Automation Use Self-hosted Agents – Provides better control over pipeline execution.Leverage HPA for CI/CD Workloads – Automatically adjust pod count during load test execution.Automate Deployment – Use Helm charts or Terraform for consistent infrastructure setup.Use CI/CD Pipelines – Automate test execution in Azure DevOps Pipelines.Optimize Cluster Cleanup – Prevent unnecessary costs by cleaning up resources after execution. 4. Automating Failure Handling & Alerts Set Up Alerting for Test Failures – Automatically detect failures in JMeter tests and trigger alerts.Send Notifications to Slack, Teams, or Email when a test fails. Example: Automated Failure Alerting YAML - script: | if grep -q "Assertion failed" test_log.jtl; then echo "Test failed! Sending alert..." curl -X POST -H "Content-Type: application/json" -d '{"text": "JMeter Test Failed! Check logs."}' <Slack_Webhook_URL> fi displayName: "Monitor & Alert for Failures" Figure 4: Automated failure detection and alerting mechanism for JMeter tests in Azure DevOps, utilizing Azure Monitor & Log Analytics for failure handling. 5. Steps for Automating JMeter Test Reporting & Email Notifications for JMeter Results Once the CI/CD pipeline generates the JTL file, we can convert it into an HTML report. Generate an HTML report from JTL: jmeter -g results.jtl -o report/ This will create a detailed performance report inside the report/ directory. Convert JTL to CSV (Optional): awk -F, '{print $1, $2, $3, $4}' results.jtl > results.csv This extracts key columns from results.jtl and saves them in results.csv. Extracting Key Metrics from JTL To summarize test results and send an email, extract key metrics like response time, error rate, and throughput. Python script to parse results.jtl and summarize key stats: Python import pandas as pd def summarize_jtl_results(jtl_file): df = pd.read_csv(jtl_file) total_requests = len(df) avg_response_time = df["elapsed"].mean() error_count = df[df["success"] == False].shape[0] error_rate = (error_count / total_requests) * 100 summary = f""" **JMeter Test Summary** --------------------------------- Total Requests: {total_requests} Avg Response Time: {avg_response_time:.2f} ms Error Count: {error_count} Error Rate: {error_rate:.2f} % --------------------------------- """ return summary # Example usage: report = summarize_jtl_results("results.jtl") print(report) Sending JMeter Reports via Email Once the report is generated, automate sending an email with the results. Python script to send JMeter reports via email: Python import smtplib import os from email.message import EmailMessage def send_email(report_file, recipient): msg = EmailMessage() msg["Subject"] = "JMeter Test Report" msg["From"] = "[email protected]" msg["To"] = recipient msg.set_content("Hi,\n\nPlease find attached the JMeter test report.\n\nBest,\nPerformance Team") with open(report_file, "rb") as file: msg.add_attachment(file.read(), maintype="application", subtype="octet-stream", filename=os.path.basename(report_file)) with smtplib.SMTP("smtp.example.com", 587) as server: server.starttls() server.login("[email protected]", "your-password") server.send_message(msg) # Example usage: send_email("report/index.html", "[email protected]") Automating the Process in CI/CD Pipeline Modify the azure-pipelines-cd.yml to Include Reporting & Emailing YAML - script: | echo "Generating JMeter Report" jmeter -g results.jtl -o report/ displayName: "Generate JMeter HTML Report" - script: | echo "Sending JMeter Report via Email" python send_email.py report/index.html [email protected] displayName: "Email JMeter Report" This ensures: The JMeter test report is generated post-execution.The report is automatically emailed to stakeholders. Conclusion By leveraging JMeter on Kubernetes and CI/CD automation with Azure DevOps (or other CI/CD tools like Jenkins, GitHub Actions, etc.), you can ensure your applications are scalable, resilient, and cost-effective. This guide covers the deployment and execution of JMeter on AKS, enabling distributed load testing at scale. By leveraging Kubernetes auto-scaling capabilities, this setup ensures efficient resource utilization and supports continuous performance testing with automated reporting and alerting. This Kubernetes-native JMeter setup allows for scalable, cost-effective, and automated performance testing on Azure Kubernetes Service (AKS) but can also be deployed on AWS EKS, Google GKE, or any other Kubernetes environment. It integrates JMeter, Kubernetes, InfluxDB, and Grafana for scalable, automated, and observable performance testing, with automated email notifications and report generation. Benefits of Automating JMeter Load Testing with CI/CD Pipelines End-to-end automation – From test execution to result storage and reporting.Scalability – JMeter runs are distributed across AKS worker nodes (or any Kubernetes cluster).Observability – Monitored via InfluxDB & Grafana with real-time insights.Automated Reporting – JTL test results are converted into HTML reports and sent via email notifications. "With modern applications handling massive traffic, performance testing is no longer optional—it's a necessity. By leveraging JMeter on Kubernetes and CI/CD automation with Azure DevOps (or any CI/CD tool), you can ensure your applications are scalable, resilient, and cost-effective." Key Takeaways: Automate Load Testing with Azure DevOps Pipelines (or Jenkins, GitHub Actions, etc.).Scale JMeter dynamically using Kubernetes & HPA across multi-cloud environments.Monitor & Analyze results with InfluxDB + Grafana in real time.Optimize Costs by using auto-scaling and scheduled tests.Enable Automated Reporting by sending test results via email notifications. Next Step: Expanding Reporting & Alerting Mechanisms in CI/CD Pipelines, including AI-based anomaly detection for performance testing and predictive failure analysis. Stay tuned for advanced insights! Take Action Today! Implement this setup in your environment—whether in Azure AKS, AWS EKS, or Google GKE—and share your feedback! References Apache JMeter - Apache JMeterTM. (n.d.). https://jmeter.apache.org/Apache JMeter - User’s Manual: Best Practices. (n.d.). https://jmeter.apache.org/usermanual/best-practices.htmlKubernetes documentation. (n.d.). Kubernetes. https://kubernetes.io/docs/Nickomang. (n.d.). Azure Kubernetes Service (AKS) documentation. Microsoft Learn. https://learn.microsoft.com/en-us/azure/aks/Chcomley. (n.d.). Azure DevOps documentation. Microsoft Learn. https://learn.microsoft.com/en-us/azure/devops/?view=azure-devopsInfluxData. (2021, December 10). InfluxDB: Open Source Time Series Database | InfluxData. https://www.influxdata.com/products/influxdb/Grafana OSS and Enterprise | Grafana documentation. (n.d.). Grafana Labs. https://grafana.com/docs/grafana/latest/Apache JMeter - User’s Manual: Generating Dashboard Report. (n.d.). https://jmeter.apache.org/usermanual/generating-dashboard.html

By Prabhu Chinnasamy
Java Enterprise Matters: Why It All Comes Back to Jakarta EE
Java Enterprise Matters: Why It All Comes Back to Jakarta EE

Enterprise Java has been a foundation for mission-critical applications for decades. From financial systems to telecom platforms, Java offers the portability, stability, and robustness needed at scale. Yet as software architecture shifts toward microservices, containers, and cloud-native paradigms, the question naturally arises: is Jakarta EE still relevant? For modern Java developers, the answer is a resounding yes. Jakarta EE provides a standardized, vendor-neutral set of APIs for building enterprise applications, and its evolution under the Eclipse Foundation has been a case study in open innovation. It bridges traditional enterprise reliability with the flexibility needed for today’s distributed systems, making it an essential tool for developers who want to build scalable, secure, and cloud-ready applications. Why Jakarta EE Is Important for Java Developers Whether you're building applications with Spring, Quarkus, or Helidon, you're relying on Jakarta EE technologies—often without even realizing it. Jakarta EE is not a competitor to these frameworks, but rather a foundational layer upon which they build. If you use Spring Data JPA, you're using the Jakarta Persistence API. If you're writing web controllers or using HTTP filters, you're relying on Jakarta Servlet under the hood. This is the core of Jakarta EE’s significance: it standardizes the APIs that power the vast majority of Java enterprise applications, regardless of the higher-level frameworks developers adopt. Spring itself had to make a sweeping change in response to Jakarta EE's namespace migration, switching from javax.* to jakarta.* packages. This "big bang" was not just a symbolic change—it was an explicit acknowledgment of Jakarta EE’s centrality in the Java ecosystem. Spring 7 is doing an upgrade on Jakarta EE Understanding Jakarta EE means understanding the base contracts that define Java enterprise development. Even if you're not programming directly against the Jakarta EE APIs, your stack is deeply intertwined with them. That's why it's essential for all Java developers, whether in legacy systems or cloud-native microservices, to be aware of Jakarta EE’s role and evolution. Jakarta EE History To understand Jakarta EE, we must first revisit Java’s architectural lineage. Java was initially split into three major platforms: Java SE (Standard Edition) for general-purpose programming, Java ME (Micro Edition) for embedded devices, and Java EE (Enterprise Edition) for large-scale server-side applications. Java EE provided a blueprint for enterprise architecture, introducing specifications such as Servlets, EJBs, JPA, and JMS that standardized application development across vendors. The Java platforms in the past Understanding Jakarta EE means understanding the base contracts that define Java enterprise development. Even if you're not programming directly against the Jakarta EE APIs, your stack is deeply intertwined with them. That's why it's essential for all Java developers, whether in legacy systems or cloud-native microservices, to be aware of Jakarta EE’s role and evolution. Jakarta EE History To understand Jakarta EE, we must first revisit Java’s architectural lineage. Java was initially split into three major platforms: Java SE (Standard Edition) for general-purpose programming, Java ME (Micro Edition) for embedded devices, and Java EE (Enterprise Edition) for large-scale server-side applications. Java EE provided a blueprint for enterprise architecture, introducing specifications such as Servlets, EJBs, JPA, and JMS that standardized application development across vendors. Eclipse Microprofile 7 Jakarta EE Platforms Jakarta EE’s flexibility is one of its biggest strengths, particularly through the profile system introduced to address different application needs. There are three official profiles: Core Profile, focused on lightweight runtimes and microservices.Web Profile, aimed at typical enterprise web applications, including Servlets, CDI, and RESTful services.Full Platform Profile, for applications that require the entire suite of Jakarta technologies like messaging, batch processing, and enterprise beans. This structured approach allows organizations to adopt only what they need, making Jakarta EE a better fit for both greenfield cloud projects and legacy modernization. Unlike frameworks that require a complete rewrite or deep vendor lock-in, Jakarta EE emphasizes interoperability and gradual evolution—an essential feature for enterprises with complex systems and long lifecycles. New and Updated Specifications in Jakarta EE 10 Jakarta EE 11—The Current Version The release of Jakarta EE 11 marks a landmark moment, as it is the first major release to break away from the legacy Java EE APIs and fully adopt the Jakarta namespace. Jakarta EE 11 introduces support for Java 17, removes outdated technologies, and aligns more closely with modern Java practices, such as modularity and sealed classes. It also introduces new specifications, such as Jakarta Data, which provides a consistent data access layer that works across various persistence mechanisms. The integration of CDI (Contexts and Dependency Injection) as the foundation of dependency management is now deeper and more streamlined across all specifications. This not only reduces boilerplate but also enables a more consistent programming model across the entire platform. Jakarta EE 11 also sets the stage for better integration with cloud-native tooling and DevOps workflows. Jakarta EE 12—The Future While Jakarta EE 11 marks a milestone of cleanup and modernization, Jakarta EE 12 looks forward to more ambitious goals. The specification is still under development, but early conversations indicate a deeper alignment with Eclipse MicroProfile, potentially integrating select MicroProfile specifications into Jakarta EE itself. This convergence would help reduce fragmentation and unify enterprise Java standards under a single umbrella. One notable change is the proposed expansion of the Web Profile, which is expected to include: Jakarta MVC for building model-view-controller-based web applications.Jakarta NoSQL for integrating NoSQL databases in a standardized way.Jakarta Data and Jakarta Query for flexible and modern data access APIs. In addition to API enhancements, Jakarta EE 12 aims to improve startup time, support native compilation, and facilitate smoother integration with technologies such as GraalVM. These improvements align Jakarta EE with the performance expectations of microservices and serverless architectures. To follow the specification’s progress, you can consult the official Jakarta EE 12 specification page. The platform’s roadmap is shaped through transparent collaboration between vendors, individuals, and organizations. The Jakarta EE Working Group and the annual Developer Survey ensure that future releases are informed by real-world feedback and community needs. Conclusion Jakarta EE may have emerged from the legacy of Java EE, but it is no longer bound by it. Instead, it offers a modern foundation for enterprise Java that respects the past while building for the future. With a robust standard, open governance, and compatibility with cloud-native innovations, Jakarta EE remains crucial for developers, companies, and the software industry as a whole. If you are a Java developer today, whether working with Spring, Quarkus, Helidon, or any other framework, you are already using Jakarta EE technologies. It powers your web applications, your data access layers, and your integration code. Jakarta EE is not something separate—it is embedded in the tools you use every day. And Jakarta EE is not driven by a single company—the community leads it. The specification process, new feature discussions, and overall roadmap are shaped by individuals and organizations who contribute because they care about the future of enterprise Java. If you rely on Jakarta EE, consider getting involved. You can follow the specifications, provide feedback, and help guide the platform's direction. It's not just a standard—it's a shared effort, and your voice matters. References Jakarta EE | Cloud Native Enterprise JAVa | Java EE | The Eclipse Foundation. (n.d.). Jakarta® EE: The New Home of Cloud Native Java. https://jakarta.ee/Jakarta EE Platform 11 (Under Development) | Jakarta EE | The Eclipse Foundation. (n.d.). Jakarta® EE: The New Home of Cloud Native Java. https://jakarta.ee/specifications/platform/11/Jakarta EE Platform 12 (Under Development) | Jakarta EE | The Eclipse Foundation. (n.d.). Jakarta® EE: The New Home of Cloud Native Java. https://jakarta.ee/specifications/platform/12MicroProfile. (2024b, December 16). Home - MicroProfile.https://microprofile.io/Obradovic, T. (n.d.). Rising Momentum in Enterprise Java: Insights from the 2024 Jakarta EE Developer Survey Report. Eclipse Foundation Staff Blogs. https://blogs.eclipse.org/post/tatjana-obradovic/rising-momentum-enterprise-java-insights-2024-jakarta-ee-developer-surveySaeed, L. (2025, February 10). 10 Ways Jakarta EE 11 modernizes Enterprise Java Development. 10 Ways Jakarta EE 11 Modernizes Enterprise Java Development.https://blog.payara.fish/10-ways-jakarta-ee-11-modernizes-enterprise-developmentMailing List: microprofile-wg (92 subscribers) | Eclipse - The Eclipse Foundation open source community website. (n.d.). https://accounts.eclipse.org/mailing-list/microprofile-wg

By Otavio Santana DZone Core CORE
Converting List to String in Terraform
Converting List to String in Terraform

In Terraform, you will often need to convert a list to a string when passing values to configurations that require a string format, such as resource names, cloud instance metadata, or labels. Terraform uses HCL (HashiCorp Configuration Language), so handling lists requires functions like join() or format(), depending on the context. How to Convert a List to a String in Terraform The join() function is the most effective way to convert a list into a string in Terraform. This concatenates list elements using a specified delimiter, making it especially useful when formatting data for use in resource names, cloud tags, or dynamically generated scripts. The join(", ", var.list_variable) function, where list_variable is the name of your list variable, merges the list elements with ", " as the separator. Here’s a simple example: Shell variable "tags" { default = ["dev", "staging", "prod"] } output "tag_list" { value = join(", ", var.tags) } The output would be: Shell "dev, staging, prod" Example 1: Formatting a Command-Line Alias for Multiple Commands In DevOps and development workflows, it’s common to run multiple commands sequentially, such as updating repositories, installing dependencies, and deploying infrastructure. Using Terraform, you can dynamically generate a shell alias that combines these commands into a single, easy-to-use shortcut. Shell variable "commands" { default = ["git pull", "npm install", "terraform apply -auto-approve"] } output "alias_command" { value = "alias deploy='${join(" && ", var.commands)}'" } Output: Shell "alias deploy='git pull && npm install && terraform apply -auto-approve'" Example 2: Creating an AWS Security Group Description Imagine you need to generate a security group rule description listing allowed ports dynamically: Shell variable "allowed_ports" { default = [22, 80, 443] } resource "aws_security_group" "example" { name = "example_sg" description = "Allowed ports: ${join(", ", [for p in var.allowed_ports : tostring(p)])}" dynamic "ingress" { for_each = var.allowed_ports content { from_port = ingress.value to_port = ingress.value protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } } } The join() function, combined with a list comprehension, generates a dynamic description like "Allowed ports: 22, 80, 443". This ensures the security group documentation remains in sync with the actual rules. Alternative Methods For most use cases, the join() function is the best choice for converting a list into a string in Terraform, but the format() and jsonencode() functions can also be useful in specific scenarios. 1. Using format() for Custom Formatting The format() function helps control the output structure while joining list items. It does not directly convert lists to strings, but it can be used in combination with join() to achieve custom formatting. Shell variable "ports" { default = [22, 80, 443] } output "formatted_ports" { value = format("Allowed ports: %s", join(" | ", var.ports)) } Output: Shell "Allowed ports: 22 | 80 | 443" 2. Using jsonencode() for JSON Output When passing structured data to APIs or Terraform modules, you can use the jsonencode() function, which converts a list into a JSON-formatted string. Shell variable "tags" { default = ["dev", "staging", "prod"] } output "json_encoded" { value = jsonencode(var.tags) } Output: Shell "["dev", "staging", "prod"]" Unlike join(), this format retains the structured array representation, which is useful for JSON-based configurations. Creating a Literal String Representation in Terraform Sometimes you need to convert a list into a literal string representation, meaning the output should preserve the exact structure as a string (e.g., including brackets, quotes, and commas like a JSON array). This is useful when passing data to APIs, logging structured information, or generating configuration files. For most cases, jsonencode() is the best option due to its structured formatting and reliability in API-related use cases. However, if you need a simple comma-separated string without additional formatting, join() is the better choice. Common Scenarios for List-to-String Conversion in Terraform Converting a list to a string in Terraform is useful in multiple scenarios where Terraform requires string values instead of lists. Here are some common use cases: Naming resources dynamically: When creating resources with names that incorporate multiple dynamic elements, such as environment, application name, and region, these components are often stored as a list for modularity. Converting them into a single string allows for consistent and descriptive naming conventions that comply with provider or organizational naming standards.Tagging infrastructure with meaningful identifiers: Tags are often key-value pairs where the value needs to be a string. If you’re tagging resources based on a list of attributes (like team names, cost centers, or project phases), converting the list into a single delimited string ensures compatibility with tagging schemas and improves downstream usability in cost analysis or inventory tools.Improving documentation via descriptions in security rules: Security groups, firewall rules, and IAM policies sometimes allow for free-form text descriptions. Providing a readable summary of a rule’s purpose, derived from a list of source services or intended users, can help operators quickly understand the intent behind the configuration without digging into implementation details.Passing variables to scripts (e.g., user_data in EC2 instances): When injecting dynamic values into startup scripts or configuration files (such as a shell script passed via user_data), you often need to convert structured data like lists into strings. This ensures the script interprets the input correctly, particularly when using loops or configuration variables derived from Terraform resources.Logging and monitoring, ensuring human-readable outputs: Terraform output values are often used for diagnostics or integration with logging/monitoring systems. Presenting a list as a human-readable string improves clarity in logs or dashboards, making it easier to audit deployments and troubleshoot issues by conveying aggregated information in a concise format. Key Points Converting lists to strings in Terraform is crucial for dynamically naming resources, structuring security group descriptions, formatting user data scripts, and generating readable logs. Using join() for readable concatenation, format() for creating formatted strings, and jsonencode() for structured output ensures clarity and consistency in Terraform configurations.

By Mariusz Michalowski
Understanding the Mandelbrot Set: A Beautiful Benchmark for Computing Power
Understanding the Mandelbrot Set: A Beautiful Benchmark for Computing Power

Imagine a mathematical object so complex that it can push computers to their limits, yet so beautiful that it has captivated mathematicians and artists alike. Welcome to the world of the Mandelbrot set—a fascinating fractal that doubles as a powerful benchmark for computing performance. Have you ever wondered how to compare the performance of different programming languages? While there are many benchmarking tools available, the Mandelbrot set offers a uniquely compelling approach that combines mathematical elegance with computational challenge. What is the Mandelbrot Set? Imagine you're embarking on a Mathematical Journey with these steps: Start with zeroSquare your current numberAdd your special "journey number"Repeat steps 2-3 many times Some journey numbers create paths that stay close to the starting point forever, while others lead to explosive growth. The Mandelbrot set is the collection of all journey numbers that keep the path within a distance of 2 from the start. Let's explore two different journeys: StepWell-Behaved Journey (0.2)Explosive Journey (2)Start0010² + 0.2 = 0.20² + 2 = 220.2² + 0.2 = 0.242² + 2 = 630.24² + 0.2 = 0.25766² + 2 = 3840.2576² + 0.2 = 0.2663538² + 2 = 1,446ResultStays close to originGrows wildlyImpactPart of Mandelbrot set (black region in visualization)Not in Mandelbrot set (colored region, color depends on growth rate) The Mathematical Definition In mathematical terms, the Mandelbrot set is defined in the complex plane. For each point c in this plane, we iterate the function f(z) = z² + c, starting with z = 0. If the absolute value of z remains bounded (typically below 2) after many iterations, the point c is in the Mandelbrot set. Here's the same example using mathematical notation: IterationWell-Behaved Point (c = 0.2)Divergent Point (c = 2)Startz₀ = 0z₀ = 01z₁ = 0² + 0.2 = 0.2z₁ = 0² + 2 = 22z₂ = 0.2² + 0.2 = 0.24z₂ = 2² + 2 = 63z₃ = 0.24² + 0.2 = 0.2576z₃ = 6² + 2 = 384z₄ = 0.2576² + 0.2 = 0.26635z₄ = 38² + 2 = 1,446ResultLikely in Mandelbrot set (|z₄| ≈ 0.26635 < 2)Not in Mandelbrot set (|z₄| = 1,446 > 2) This mathematical representation shows how the same process defines the Mandelbrot set, creating a bridge between the intuitive journey analogy and the formal mathematical concept. Visualization Example To help visualize what we’re discussing, I’ve created an interactive Mandelbrot set Visualization Playground. You can explore the fractal in real-time and find the complete implementation in my GitHub repository: This visualization demonstrates the concepts we've discussed: Black region: Points that stay bounded (the Mandelbrot set)Colored regions: Points that grow beyond bounds, with colors indicating growth speedBlue/Purple: Slower growthRed/Yellow: Faster growth Creating the Visual Magic The stunning visuals of the Mandelbrot set come from how we color each pixel based on its calculation results. Here's a quick note on the mathematics behind it: The set is bounded within (-2.5, 1.0) on the x-axis and (-1.0, 1.0) on the y-axis because these boundaries capture all points that stay bounded under iteration.If a sequence stays bounded (below 2) after many iterations, that point is part of the Mandelbrot set and appears black in our visualization.All other points are colored based on how quickly their sequences grow beyond this boundary. Let's break down the coloring process with a practical example using an 800x600 image: Coordinate Mapping: Each pixel maps to a unique point in the Mandelbrot space: Python # The Mandelbrot set exists within these mathematical boundaries: # x-axis: from -2.5 to 1.0 (total range of 3.5) # y-axis: from -1.0 to 1.0 (total range of 2.0) # For pixel (400, 300) in an 800x600 image: x_coordinate = (400/800) * 3.5 - 2.5 # Maps 0-800 to -2.5 to 1.0 y_coordinate = (300/600) * 2.0 - 1.0 # Maps 0-600 to -1.0 to 1.0 Color Assignment: Python def get_color(iteration_count, max_iter=100): # Points that stay bounded (part of Mandelbrot set) if iteration_count == max_iter: return (0, 0, 0) # Black # Convert iteration count to a percentage (0-100%) percentage = (iteration_count * 100) // max_iter # Map percentage to color ranges (0-255) # First third (0-33%): Blue dominates if percentage < 33: blue = 255 red = percentage * 7 # Gradually increase red return (red, 0, blue) # Example: (70, 0, 255) for 10% # Second third (33-66%): Purple to red transition elif percentage < 66: blue = 255 - ((percentage - 33) * 7) # Decrease blue return (255, 0, blue) # Example: (255, 0, 128) for 50% # Final third (66-100%): Red to yellow transition else: green = (percentage - 66) * 7 # Increase green return (255, green, 0) # Example: (255, 128, 0) for 83% Why Is It Computationally Intensive? Creating a visual representation of the Mandelbrot set requires performing these calculations for every pixel in an image. This is computationally demanding because: Volume of Calculations: A typical HD image has over 2 million pixels, each requiring its own sequence of calculations with hundreds or thousands of iterations.Precision Matters: Small numerical errors can lead to completely different results, necessitating high-precision arithmetic to maintain the fractal's integrity.No Shortcuts: Each pixel's calculation is independent and must be fully computed, as we can't predict if a sequence will stay bounded without performing the iterations. Using the Mandelbrot Set as a Benchmark The Mandelbrot set is an excellent choice for comparing programming language performance for several reasons: Easily Comparable: The output should be identical across all implementations, making verification straightforward through visual comparison.Scalable Complexity: Adjustable parameters for image size and iteration count create varying workloads for different testing scenarios.Tests Multiple Aspects: It evaluates floating-point arithmetic performance, loop optimization, memory management, and potential for parallelization. Implementation for Benchmarking Here's a basic structure that you could implement in any programming language: def calculate_mandelbrot(width, height, max_iter): # Convert each pixel coordinate to Mandelbrot space for y in range(height): for x in range(width): # Map pixel to mathematical coordinate x0 = (x / width) * 3.5 - 2.5 y0 = (y / height) * 2.0 - 1.0 # Initialize values xi = yi = iter = 0 # Main calculation loop while iter < max_iter and (xi * xi + yi * yi) <= 4: # Calculate next values tmp = xi * xi - yi * yi + x0 yi = 2 * xi * yi + y0 xi = tmp iter += 1 # Color pixel based on iteration count result[y * width + x] = iter Tips for Effective Benchmarking When using the Mandelbrot set as a benchmark: Parameter Standardization: Establish uniform test conditions by fixing image dimensions, iteration limits, and precision requirements across all benchmark implementations.Performance Metrics: Comprehensively evaluate performance by measuring calculation time, memory consumption, and computational resource utilization.Holistic Evaluation: Assess not just technical performance, but also implementation complexity, code quality, and optimization potential across different programming environments. Conclusion The Mandelbrot set isn't just a mathematically fascinating object—it's also a powerful tool for comparing programming language performance. Its combination of intensive computation, precision requirements, and visual verification makes it an ideal benchmark candidate. Whether you're choosing a language for a performance-critical project or just curious about relative language speeds, implementing the Mandelbrot set can provide valuable insights into computational efficiency. The most effective benchmark aligns with your specific use case. While the Mandelbrot set excels at measuring raw computational power and floating-point performance, supplementing it with additional benchmarks will provide a more complete picture of language performance for your particular needs.

By Vivek Vellaiyappan Surulimuthu
How Node.js Works Behind the Scenes (HTTP, Libuv, and Event Emitters)
How Node.js Works Behind the Scenes (HTTP, Libuv, and Event Emitters)

When working with Node.js, most people just learn how to use it to build apps or run servers—but very few stop to ask how it actually works under the hood. Understanding the inner workings of Node.js helps you write better, more efficient code. It also makes debugging and optimizing your apps much easier. A lot of developers think Node.js is just "JavaScript with server features". That’s not entirely true. While it uses JavaScript, Node.js is much more than that. It includes powerful tools and libraries that give JavaScript abilities it normally doesn't have—like accessing your computer’s file system or handling network requests. These extra powers come from something deeper happening behind the scenes, and that's what this blog will help you understand. Setting the Stage: A Simple HTTP Server Before we dive into the internals of Node.js, let’s build something simple and useful—a basic web server using only Node.js, without using popular frameworks like Express. This will help us understand what’s happening behind the scenes when a Node.js server handles a web request. What Is a Web Server? A web server is a program that listens for requests from users (like opening a website) and sends back responses (like the HTML content of that page). In Node.js, we can build such a server in just a few lines of code. Introducing the http Module Node.js comes with built-in modules—these are tools that are part of Node itself. One of them is the http module. It allows Node.js to create servers and handle HTTP requests and responses. To use it, we first need to import it into our file. JavaScript const http = require('http'); This line gives us access to everything the http module can do. Creating a Basic Server Now let’s create a very simple server: JavaScript const http = require('http'); const server = http.createServer((request, response) => { response.statusCode = 200; // Status 200 means 'OK' response.setHeader('Content-Type', 'text/plain'); // Tell the browser what we are sending response.end('Hello, World!'); // End the response and send 'Hello, World!' to the client }); server.listen(4000, () => { console.log('Server is running on http://localhost:4000'); }); What Does This Code Do? http.createServer() creates the server.It takes a function as an argument. This function runs every time someone makes a request to the server.This function has two parameters: request:contains info about what the user is asking for. response: lets us decide what to send back. Let’s break it down even more: response Object This object has data like: The URL the user is visiting (request.url)The method they are using (GET, POST, etc.) (request.method)The headers (browser info, cookies, etc.) response Object This object lets us: Set the status code (e.g., 200 OK, 404 Not Found)Set headers (e.g., Content-Type: JSON, HTML, etc.)Send a message back using .end() The Real Story: Behind the HTTP Module At first glance, it looks like JavaScript can do everything: create servers, read files, talk to the internet. But here’s the truth... JavaScript alone can't do any of that. Let’s break this down. What JavaScript Can’t Do Alone JavaScript was originally made to run inside web browsers—to add interactivity to websites. Inside a browser, JavaScript doesn’t have permission to: Access your computer’s filesTalk directly to the network (like creating a server)Listen on a port (like port 4000) Browsers protect users by not allowing JavaScript to access low-level features like the file system or network interfaces. So if JavaScript can’t do it... how is Node.js doing it? Enter Node.js: The Bridge Between JS and Your System Node.js gives JavaScript superpowers by using system-level modules written in C and C++ under the hood. These modules give JavaScript access to your computer’s core features. Let’s take the http module as an example. When you write: JavaScript const http = require('http'); You're not using pure JavaScript. You're actually using a Node.js wrapper that connects JavaScript to C/C++ libraries in the background. What Does the http Module Really Do? The http module: Uses C/C++ code under the hood to access the network interface (something JavaScript alone can't do).Wraps all that complexity into a JavaScript-friendly format.Exposes simple functions like createServer() and methods like request.end(). Think of it like this: Your JavaScript is the user-friendly remoteNode.js modules are the wires and electronics inside the machine You write friendly code, but Node does the heavy lifting using system-level access. Proof: JavaScript Can’t Create a Server on Its Own Try running this in the browser console: JavaScript const http = require('http'); You’ll get an error: require is not defined. That’s because require and the http module don’t exist in browsers. They are Node.js features, not JavaScript features. Real-World Example: What’s Actually Happening Let’s go back to our previous server code: JavaScript const http = require('http'); const server = http.createServer((req, res) => { res.statusCode = 200; res.setHeader('Content-Type', 'text/plain'); res.end('Hello from Node!'); }); server.listen(4000, () => { console.log('Server is listening on port 4000'); }); What’s really happening here? require('http') loads a Node.js module that connects to your computer’s network card using libuv (a C library under Node).createServer() sets up event listeners for incoming requests on your computer’s port.When someone visits http://localhost:4000, Node.js receives that request and passes it to your JavaScript code. JavaScript decides what to do using the req and res objects. Why This Matters Once you understand that JavaScript is only part of the picture, you’ll write smarter code. You’ll realize: Why Node.js has modules like fs (file system), http, crypto, etc.Why these modules feel more powerful than regular JavaScript—they are.That Node.js is really a layer that connects JavaScript to the operating system. In short: JavaScript can’t talk to the system. Node.js can—and it lets JavaScript borrow that power. The Role of Libuv So far, we’ve seen that JavaScript alone can't do things like network access or reading files. Node.js solves that by giving us modules like http, fs, and more. But there’s something even deeper making all of this work: a powerful C library called libuv. Let’s unpack what libuv is, what it does, and why it’s so important. What Is Libuv? Libuv is a C-based library that handles all the low-level, operating system tasks that JavaScript can't touch. Think of it like this: Libuv is the engine under the hood of Node.js. It handles the tough system-level jobs like: Managing filesManaging networksHandling threads (multi-tasking)Keeping track of timers and async tasks Why Node.js Needs Libuv JavaScript is single-threaded—meaning it can only do one thing at a time. But in real-world apps, you need to do many things at once, like: Accept web requestsRead/write to filesCall APIsWait for user input If JavaScript did all of these tasks by itself, it would block everything else and slow down your app. This is where libuv saves the day. Libuv takes those slow tasks, runs them in the background, and lets JavaScript move on. When the background task is done, libuv sends the result back to JavaScript. How Libuv Acts as a Bridge Here’s what happens when someone sends a request to your Node.js server: The request hits your computer’s network interface.Libuv detects this request.It wraps it into an event that JavaScript can understand.Node.js triggers your callback (your function with request and response).Your JavaScript code runs and responds to the user. You didn’t have to manually manage threads or low-level sockets—libuv took care of it. A Visual Mental Model (Simplified) Client (Browser) ↓ Operating System (receives request) ↓ Libuv (converts it to a JS event) ↓ Node.js (runs your JavaScript function) Real-World Analogy: Restaurant Imagine JavaScript as a chef with one hand. He can only cook one dish at a time. Libuv is like a kitchen assistant who: Takes orders from customersGets ingredients readyTurns on the stoveRings a bell when the chef should jump in Thanks to libuv, the chef stays focused and fast, while the assistant takes care of background tasks. Behind-the-Scenes Example Let’s say you write this Node.js code: JavaScript const fs = require('fs'); fs.readFile('myfile.txt', 'utf8', (err, data) => { if (err) throw err; console.log('File content:', data); }); console.log('Reading file...'); What happens here: fs.readFile() is not handled by JavaScript alone.Libuv takes over, reads the file in the background.Meanwhile, "Reading file..." prints immediately.When the file is ready, libuv emits an event.Your callback runs and prints the file content. Output: Reading file... File content: Hello from the file! Libuv Handles More Than Files Libuv also manages: Network requests (like your HTTP server)Timers (setTimeout, setInterval)DNS lookupsChild processesSignals and Events Basically, everything async and powerful in Node.js is powered by libuv. Summary: Why Libuv Matters Libuv makes Node.js non-blocking and fast.It bridges JavaScript with system-level features (network, file, threads).It handles background work, then notifies JavaScript when ready.Without libuv, Node.js would be just JavaScript—and very limited. Breaking Down Request and Response When you create a web server in Node.js, you always get two special objects in your callback function: request and response. Let’s break them down so you understand what they are, how they work, and why they’re important. The Basics Here’s a sample server again: JavaScript const http = require('http'); const server = http.createServer((request, response) => { // We'll explain what request and response do in a moment }); server.listen(4000, () => { console.log('Server running at http://localhost:4000'); }); Every time someone visits your server, Node.js runs the callback you gave to createServer(). That callback automatically receives two arguments: request: contains all the info about what the client is asking for.response: lets you send back the reply. What Is request? The request object is an instance of IncomingMessage. That means it’s a special object that contains properties describing the incoming request. Here’s what you can get from it: JavaScript http.createServer((req, res) => { console.log('Method:', req.method); // e.g., GET, POST console.log('URL:', req.url); // e.g., /home, /about console.log('Headers:', req.headers); // browser info, cookies, etc. res.end('Request received'); }); Common use cases: req.method: What type of request is it? (GET, POST, etc.)req.url: Which page or resource is being requested?req.headers: Metadata about the request (browser type, accepted content types, etc.) What Is response? The response object is an instance of ServerResponse. That means it comes with many methods you can use to build your reply. Here’s a basic usage: JavaScript http.createServer((req, res) => { res.statusCode = 200; // OK res.setHeader('Content-Type', 'text/plain'); res.end('Hello, this is your response!'); }); Key methods and properties: res.statusCode: Set the HTTP status (e.g., 200 OK, 404 Not Found)res.setHeader(): Set response headers like content typeres.end(): Ends the response and sends it to the client Streams in Response Node.js is built around the idea of streams—data that flows bit by bit. The response object is actually a writable stream. That means you can: Write data in chunks (res.write(data))End the response with res.end() JavaScript http.createServer((req, res) => { res.write('Step 1\n'); res.write('Step 2\n'); res.end('All done!\n'); // Closes the stream }); Why is this useful? In large apps, data might not be ready all at once (like fetching from a database). Streams let you send parts of the response as they are ready, which improves performance. Behind the Scenes: Automatic Injection You don’t create request and response manually. Node.js does it for you automatically. Think of it like this: A user visits your site.Node.js uses libuv to detect the request.It creates request and response objects.It passes them into your server function like magic: JavaScript http.createServer((request, response) => { // Node.js gave you these to work with }); You just catch them in your function and use them however you need. Recap: Key Differences Object Type Used For Main Features request IncomingMessage Reading data from client Properties like .method, .url response ServerResponse Sending data to the client Methods like .write(), .end() Example: A Tiny Routing Server Let’s put it all together: JavaScript const http = require('http'); http.createServer((req, res) => { if (req.url === '/hello') { res.statusCode = 200; res.setHeader('Content-Type', 'text/plain'); res.end('Hello there!'); } else { res.statusCode = 404; res.end('Page not found'); } }).listen(4000, () => { console.log('Server is running at http://localhost:4000'); }); This code: Reads req.urlSends a custom response using res.end()Demonstrates how Node.js handles different routes without Express Final Thought Every time you use request and response in Node.js, you're working with powerful objects that represent real-time communication over the internet. These objects are the foundation of building web servers and real-time applications using Node.js. Once you understand how they work, developing scalable and responsive apps becomes much easier. Event Emitters and Execution Flow Node.js is famous for being fast and efficient—even though it uses a single thread (one task at a time). So how does it manage to handle thousands of requests without slowing down? The secret lies in how Node.js uses events to control the flow of code execution. Let’s explore how this works behind the scenes. What Is an Event? An event is something that happens. For example: A user visits your website → that’s a request eventA file finishes loading → that’s a file eventA timer runs out → that’s a timer event Node.js watches for these events and runs your code only when needed. What Is an Event Emitter? An EventEmitter is a tool in Node.js that: Listens for a specific eventRuns a function (handler) when that event happens It’s like a doorbell: You push the button → an event happensThe bell rings → a function gets triggered How Node.js Handles a Request with Events Let’s revisit our HTTP server: JavaScript const http = require('http'); const server = http.createServer((req, res) => { res.end('Hello from Node!'); }); server.listen(4000, () => { console.log('Server is running...'); }); Here’s what’s really happening: You start the server with server.listen()Node.js waits silently—no code inside createServer() runs yetWhen someone visits http://localhost:4000, that triggers a request eventNode.js emits the request eventYour callback ((req, res) => { ... }) runs only when that event happens That’s event-driven programming in action. EventEmitter in Action (Custom Example) You can create your own events using Node’s built-in events module: JavaScript const EventEmitter = require('events'); const myEmitter = new EventEmitter(); // Register an event handler myEmitter.on('greet', () => { console.log('Hello there!'); }); // Emit the event myEmitter.emit('greet'); // Output: Hello there! This is the same system that powers http.createServer(). Internally, it uses EventEmitter to wait for and handle incoming requests. Why Node.js Waits for Events Node.js is single-threaded, meaning it only runs one task at a time. But thanks to libuv and event emitters, it can handle tasks asynchronously without blocking the thread. Here’s what that means: JavaScript const fs = require('fs'); fs.readFile('file.txt', 'utf8', (err, data) => { console.log('File read complete!'); }); console.log('Reading file...'); Output: Reading file... File read complete! Even though reading the file takes time, Node doesn’t wait. It moves on, and the file event handler runs later when the file is ready. Role of Routes and Memory in Execution Flow Let’s say your server handles three routes: JavaScript const http = require('http'); http.createServer((req, res) => { if (req.url === '/') { res.end('Home Page'); } else if (req.url === '/about') { res.end('About Page'); } else { res.end('404 Not Found'); } }).listen(4000); Node.js keeps all these routes in memory, but none of them run right away. They only run: When a matching URL is requestedAnd the event for that request is emitted That’s why Node.js is efficient—it doesn't waste time running unnecessary code. How This Helps You Understanding this model lets you: Write non-blocking, scalable applicationsAvoid unnecessary code executionStructure your apps better (especially using frameworks like Express) Recap: Key Concepts Concept Role Event Something that happens (e.g. a request, a timer) EventEmitter Node.js feature that listens for and reacts to events createServer() Registers a handler for request events Execution Flow Code only runs after the relevant event occurs Single Thread Node.js uses one thread but handles many tasks using events & libuv The Real Mental Model of Node.js To truly understand how Node.js works behind the scenes, think of it as a layered system, like an onion. Each layer has a role—and together, they turn a simple user request into working JavaScript code. Let’s break it down step by step using this flow: Layered Flow: Client → OS → Libuv → Node.js → JavaScript 1. Client (The User’s Browser or App) Everything starts when a user does something—like opening a web page or clicking a button. This action sends a request to your server. Example: When a user opens http://localhost:4000/hello, their browser sends a request to port 4000 on your computer. 2. OS (Operating System) The request first hits your computer’s operating system (Windows, Linux, macOS). The OS checks: What port is this request trying to reach?Is there any application listening on that port? If yes, it passes the request to that application—in this case, your Node.js server. 3. Libuv (The Bridge Layer) Here’s where libuv takes over. This powerful library does the dirty work: It listens to system-level events like network activityIt detects the incoming request from the OSIt creates internal event objects (like “a request just arrived”) Libuv doesn't handle the request directly—it simply prepares it and signals Node.js: “Hey, a new request is here!” 4. Node.js (The Runtime) Node.js receives the event from libuv and emits a request event. Now, Node looks for a function you wrote that listens for that event. For HTTP servers, this is the function you passed to http.createServer(): JavaScript const server = http.createServer((req, res) => { // This runs when the 'request' event is triggered }); Here, Node.js automatically injects two objects: req = details about the incoming request res = tools to build and send a response You didn’t create these objects—they were passed in by Node.js, based on the info that came from libuv. 5. JavaScript (Your Logic) Now it's your turn. With the req and res objects in hand, your JavaScript code finally runs: JavaScript const http = require('http'); const server = http.createServer((req, res) => { if (req.url === '/hello') { res.statusCode = 200; res.end('Hello from Node.js!'); } else { res.statusCode = 404; res.end('Not found'); } }); server.listen(4000, () => { console.log('Server is ready on port 4000'); }); All this logic sits at the final layer—the JavaScript layer. But none of it happens until the earlier layers do their job. Diagram: How a Request Is Handled Here’s a simple text-based version of the diagram: [ Client ] ↓ [ Operating System ] ↓ [ libuv (C library) ] ↓ [ Node.js runtime (Event emitters, APIs) ] ↓ [ Your JavaScript function (req, res) ] Each layer processes the request a bit and passes it along, until your code finally decides how to respond. Why This Mental Model Matters Most developers only think in terms of JavaScript. But when you understand the whole flow: You can troubleshoot issues better (e.g., why your server isn’t responding)You realize the real power behind Node.js isn't JavaScript—it’s how Node connects JS to the systemYou appreciate libraries like Express, which simplify this flow for you Recap: What Happens on Each Layer Layer Responsibility Client Sends HTTP request OS Receives the request and passes it to the correct app Libuv Listens for the request, creates an event Node.js Emits the event, injects req and res, runs server JavaScript Uses req and res to handle and send a response This mental model helps you see Node.js not just as "JavaScript for servers," but as a powerful system that turns low-level OS events into high-level JavaScript code. Conclusion Node.js may look like just JavaScript for the server, but it’s much more than that. It’s a powerful system that connects JavaScript to your computer’s core features using C/C++ libraries like libuv. While you focus on writing logic, Node handles the hard work—managing files, networks, and background tasks. Even the simplest server code runs on top of a smart and complex architecture. Understanding what happens behind the scenes helps you write better, faster, and more reliable applications.

By Sanjay Singhania
Building Generative AI Services: An Introductory and Practical Guide
Building Generative AI Services: An Introductory and Practical Guide

Amazon Web Services (AWS) offers a vast range of generative artificial intelligence solutions, which allow developers to add advanced AI capabilities to their applications without having to worry about the underlying infrastructure. This report highlights the creation of functional applications using Amazon Bedrock, which is a serverless offering based on an API that provides access to core models from leading suppliers, including Anthropic, Stability AI, and Amazon. As the demand for AI-powered applications grows, developers seek easy and scalable solutions to integrate generative AI into their applications. AWS provides this capability through the firm's proprietary generative AI services, and the standout among these is Amazon Bedrock. Amazon Bedrock enables you to access foundation models via API without worrying about underlying infrastructure, scaling, and model training. Through this practical guide, you learn how to utilize Bedrock to achieve a variety of generation tasks, including Q&A, summarization, image generation, conversational AI, and semantic search. Local Environment Setup Let's get started by setting up the AWS SDK for Python and configuring our AWS credentials. Shell pip install boto3 aws configure Confirm that your account has access to the Bedrock service and underlying foundation models via the AWS console. Once done, we can experiment with some generative AI use cases! Intelligent Q&A With Claude v2 The current application demonstrates how one can create a question-and-answer assistant using the Anthropic model v2. Forming the input as a conversation allows you to instruct the assistant to give concise, on-topic answers to user questions. Such an application is especially ideal for customer service, knowledge bases, or virtual helpdesk agents. Let's take a look at a practical example of talking with Claude: Python import boto3 import json client = boto3.client("bedrock-runtime", region_name="us-east-1") body = { "prompt": "Human: How can I reset my password?\n\nAssistant:", "max_tokens_to_sample": 200, "temperature": 0.7, "stop_sequences": ["\nHuman:"] } response = client.invoke_model( modelId="anthropic.claude-v2", contentType="application/json", accept="application/json", body=json.dumps(body) ) print(response['body'].read().decode()) This prompt category simulates a human question while a knowledgeable assistant gives structured and coherent answers. A variation of this method can be utilized to create custom assistants that provide logically correct responses to user queries. Summarization Using Amazon Titan Amazon Titan text model enables easy summarization of long texts to concise and meaningful abstractions. Amazon Titan text model greatly improves the reading experience, enhances user engagement, and minimizes cognitive loads for such applications as news reporting, legal documents, and research papers. Python body = { "inputText": "Cloud computing provides scalable IT resources via the internet...", "taskType": "summarize" } response = client.invoke_model( modelId="amazon.titan-text-lite-v1", contentType="application/json", accept="application/json", body=json.dumps(body) ) print(response['body'].read().decode()) By altering the nature of the task and the source text, we can implement the same strategy in content simplification, keyword extraction, and paraphrasing. Text-to-Image Generation Using Stability AI Visual content is crucial to marketing, social media, and product design. Using Stability AI's Stable Diffusion model in Bedrock, a user can generate images from text prompts, thus simplifying creative workflows or enabling real-time content generation features. Python import base64 from PIL import Image from io import BytesIO body = { "prompt": "A futuristic smart ring with a holographic display on a table", "cfg_scale": 10, "steps": 50 } response = client.invoke_model( modelId="stability.stable-diffusion-xl-v0", contentType="application/json", accept="application/json", body=json.dumps(body) ) image_data = json.loads(response['body'].read()) img_bytes = base64.b64decode(image_data['artifacts'][0]['base64']) Image.open(BytesIO(img_bytes)).show() This technique is especially well-adapted to user interface mockups, game industry asset production, or real-time visualization tools in design software. Conversation With Claude v2 Let's expand on the Q&A example. For example, this sample use case demonstrates a sample multi-turn conversation experience in Claude v2. The assistant maintains context and answers properly through conversational steps: Python conversation = """ Human: Help me plan a trip to Seattle. Assistant: Sure! Business or leisure? Human: Leisure. Assistant: """ body = { "prompt": conversation, "max_tokens_to_sample": 200, "temperature": 0.5, "stop_sequences": ["\nHuman:"] } response = client.invoke_model( modelId="anthropic.claude-v2", contentType="application/json", accept="application/json", body=json.dumps(body) ) print(response['body'].read().decode() Interacting in multi-turn conversations is crucial for building booking agents, chatbots, or any agent that is meant to gather sequential information from users. Using Embeddings for Retrieval Text embeddings are quantitative representations containing semantic meaning. Amazon Titan generates embeddings that can be stored in vector databases to be used in semantic search, recommendation systems, or similarity measurement. Python body = { "inputText": "Explain zero trust architecture." } response = client.invoke_model( modelId="amazon.titan-embed-text-v1", contentType="application/json", accept="application/json", body=json.dumps(body) ) embedding_vector = json.loads(response["body".read()])['embedding'] print(len(embedding_vector)) You can retrieve documents by meaning using embeddings, which greatly improves retrieval efficiency for consumer and enterprise applications. Additional Day-to-Day Applications By integrating these important usage scenarios, developers can build well-architected production-grade applications. For example: A customer service system can make use of Claude to interact in question-and-answer conversations, utilize Titan to summarize content, and employ embeddings to search for documents.A design application can utilize Stable Diffusion to generate images based on user-defined parameters.A bot driven by Claude can escalate requests to the human through AWS Lambda functions in the bot. AWS Bedrock provides out-of-box integration for services including Amazon Kendra (enterprise search across documents), AWS Lambda (serverless backend functionality), and Amazon API Gateway (scalable APIs) to enable full-stack generative applications. Conclusion Generative AI services from AWS, especially Amazon Bedrock, provide developers with versatile, scalable tools to implement advanced AI use cases with ease. By using serverless APIs to invoke text, image, and embedding models, you can accelerate product development without managing model infrastructure. Whether building assistants, summarizers, generators, or search engines, Bedrock delivers enterprise-grade performance and simplicity.

By Srinivas Chippagiri DZone Core CORE

The Latest Coding Topics

article thumbnail
Building an IoT Framework: Essential Components for Success
Understanding the building blocks for creating secure, scalable, and efficient IoT frameworks that deliver real-world value and future-proof performance.
June 20, 2025
by Carsten Rhod Gregersen
· 610 Views
article thumbnail
Your Kubernetes Survival Kit: Master Observability, Security, and Automation
Master Kubernetes with this guide to observability (Tracestore), security (OPA), automation (Flagger), and custom metrics. Includes Java/Node.js examples.
June 20, 2025
by Prabhu Chinnasamy
· 704 Views · 22 Likes
article thumbnail
How to Marry MDC With Spring Integration
This article explores the challenges of Mapped Diagnostic Context propagation in Spring integration to ensure the correct context persists across workflows.
June 20, 2025
by Vsevolod Vasilyev
· 858 Views
article thumbnail
The Scrum Guide Expansion Pack
While attempting to cure Scrum’s reputation crisis, the Scrum Guide Learn how Expansion Pack may actually amplify the very problems it seeks to solve.
June 19, 2025
by Stefan Wolpers DZone Core CORE
· 744 Views
article thumbnail
Beyond Automation: How Artificial Intelligence Is Transforming Software Development
AI is more than a tool, it’s a teammate. See how it’s helping developers move faster, tackle tough problems, and focus more on building great software.
June 19, 2025
by SAURABH AGARWAL
· 958 Views · 8 Likes
article thumbnail
Complete Guide: Managing Multiple GitHub Accounts on One System
This guide helps you configure and manage multiple GitHub accounts on the same system (e.g., personal and work accounts).
June 19, 2025
by Prathap Reddy M
· 921 Views · 1 Like
article thumbnail
How Docker Desktop Enhances Developer Workflows and Observability
Docker Desktop offers a UI for local container development, with tools for Kubernetes, monitoring, and resource management.
June 19, 2025
by Ravi Teja Thutari
· 905 Views · 1 Like
article thumbnail
TIOBE Index for June 2025: Top 10 Most Popular Programming Languages
In the TIOBE Programming Community Index for June 2025, SQL has dropped out of the top 10, and the classic programming language Fortran took its place.
June 19, 2025
by Megan Crouse
· 1,068 Views
article thumbnail
How to Add a Jenkins Agent With Docker Compose
A comprehensive step-by-step tutorial to add a Jenkins agent using Docker Compose. Simplify CI/CD setup with this step-by-step guide for scalable automation.
June 19, 2025
by Faisal Khatri DZone Core CORE
· 888 Views
article thumbnail
TIOBE Programming Index News June 2025: SQL Falls to Record Low Popularity
Based on the June 2025 TIOBE Programming Community Index, SQL has faded in popularity as people working in the field of AI switch to NoSQL databases.
June 18, 2025
by Megan Crouse
· 849 Views
article thumbnail
Top Trends for Data Streaming With Apache Kafka and Flink
Explore how Apache Kafka and Apache Flink are transforming data streaming, powering real-time analytics, and shaping cloud and future-ready business systems.
June 18, 2025
by Kai Wähner DZone Core CORE
· 1,123 Views
article thumbnail
Why Whole-Document Sentiment Analysis Fails and How Section-Level Scoring Fixes It
Discover why whole-document sentiment analysis falls short and how a new open-source Python package solves it with section-level scoring.
June 18, 2025
by Sanjay Krishnegowda
· 861 Views
article thumbnail
Effective Exception Handling in Java and Spring Boot Applications
Centralize your error handling using @ControllerAdvice and @ExceptionHandler to ensure consistent, maintainable exception management across your Spring Boot application.
June 17, 2025
by Arunkumar Kallyodan
· 2,888 Views · 2 Likes
article thumbnail
Elevating LLMs With Tool Use: A Simple Agentic Framework Using LangChain
Build a smart agent with LangChain that allows LLMs to look for the latest trends, search the web, and summarize results using real-time tool calling.
June 17, 2025
by Arjun Bali
· 927 Views
article thumbnail
Build Your Private Cloud at Home
OpenStack is a popular open-source cloud platform. Through this article, I plan to provide a guide for creating a basic OpenStack cloud at home.
June 17, 2025
by Amith Kotu
· 1,155 Views · 2 Likes
article thumbnail
Kung Fu Commands: Shifu Teaches Po the Command Pattern with Java Functional Interfaces
Po skips training (again) but learns the Command Pattern from Shifu instead. Part of our "Design Patterns with Kung Fu" series. No dumplings were harmed.
June 17, 2025
by Shamik Mitra
· 842 Views
article thumbnail
How to Achieve SOC 2 Compliance in AWS Cloud Environments
Achieving SOC 2 compliance in AWS requires planning, rigorous implementation, and ongoing commitment to security best practices.
June 17, 2025
by Chase Bolt
· 849 Views
article thumbnail
AI's Cognitive Cost: How Over-Reliance on AI Tools Impacts Critical Thinking
Learn how over-reliance on AI tools impacts critical thinking, with insights from Michael Gerlich's 2025 study on cognitive offloading and AI usage trends.
June 17, 2025
by Srinivas Chippagiri DZone Core CORE
· 646 Views · 1 Like
article thumbnail
Driving Streaming Intelligence On-Premises: Real-Time ML With Apache Kafka and Flink
This article explores how to design, build, and deploy a predictive ML model using Flink and Kafka in an on-premises environment to power real-time analytics.
June 17, 2025
by Gautam Goswami DZone Core CORE
· 854 Views
article thumbnail
How to Use Testcontainers With ScyllaDB
Learn how to use Testcontainers to create lightweight, throwaway instances of ScyllaDB for testing with hands-on example.
June 16, 2025
by Eduard Knezovic
· 901 Views
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • ...
  • Next

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: