DZone Spotlight

MuleSoft is a powerful integration platform that often deals with high-throughput workloads that require robust database connection management. One solution that stands out in optimizing database interactions is HikariCP, a high-performance JDBC connection pool known for its speed and reliability. HikariCP is widely used in applications that require efficient connection management. In this article, we'll discuss the integration of HikariCP with MuleSoft, its benefits, and best practices for configuring it to maximize performance. What Is HikariCP? HikariCP is a lightweight, high-performance JDBC connection pool known for its minimal overhead and advanced optimization features. It provides fast connection acquisition, efficient connection pool management, and built-in tools to reduce latency and improve database interaction reliability. Key Features High Performance: Low-latency operations and efficient resource utilization make it one of the fastest connection pools available.Reliability: Features like connection validation and leak detection enhance stability.Scalability: Supports high-concurrency applications with minimal thread contention.Lightweight: Minimal footprint in terms of memory and CPU usage. Why Use HikariCP in MuleSoft? MuleSoft applications often interact with databases to process real-time and batch data. Efficient management of database connections is critical to meeting high transaction per minute (TPM) requirements and ensuring system reliability. HikariCP offers: Faster Response Times: Reduced connection acquisition time leads to lower latency in API responses.Enhanced Throughput: Optimized connection pooling ensures better handling of concurrent requests.Thread Management: Prevents thread saturation and reduces CPU overhead.Error Handling: Automatically detects and manages problematic connections, reducing application downtime. Integrating HikariCP With MuleSoft To integrate HikariCP in MuleSoft, follow these steps: Step 1: Configure the HikariCP Module MuleSoft does not natively include HikariCP (it comes with C3P0 by default), it has to be added as a custom dependency. Update your project's pom.xml to include the HikariCP library: XML <dependency> <groupId>com.zaxxer</groupId> <artifactId>HikariCP</artifactId> <version>5.0.1</version> <!-- Use the latest stable version --> </dependency> Step 2: Define HikariCP Configuration Add Spring dependencies in pom.xml. The easiest way to add Spring jars is via "Spring Authorization Filter" from the Mule palette: XML <dependency> <groupId>org.mule.modules</groupId> <artifactId>mule-spring-module</artifactId> <version>1.3.6</version> <classifier>mule-plugin</classifier> </dependency> <dependency> <groupId>org.springframework.security</groupId> <artifactId>spring-security-core</artifactId> <version>5.4.2</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-beans</artifactId> <version>5.3.2</version> </dependency> <dependency> <groupId>org.springframework.security</groupId> <artifactId>spring-security-config</artifactId> <version>5.4.2</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-context</artifactId> <version>5.3.2</version> </dependency> <dependency> <groupId>org.springframework</groupId> <artifactId>spring-core</artifactId> <version>5.3.2</version> </dependency> Step 3: Create Spring Configuration Define a Spring configuration XML file (spring-config.xml) to initialize the HikariCP DataSource and expose it as a Spring Bean. Add this XML config file to src/main/resources. XML <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd"> <!-- HikariCP DataSource Configuration --> <bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource" destroy-method="close"> <property name="jdbcUrl" value="jdbc:mysql://localhost:3306/mydb"/> <property name="username" value="dbuser"/> <property name="password" value="dbpassword"/> <property name="maximumPoolSize" value="20"/> <property name="minimumIdle" value="5"/> <property name="idleTimeout" value="30000"/> <property name="connectionTimeout" value="30000"/> <property name="maxLifetime" value="1800000"/> <property name="poolName" value="MuleHikariCP"/> </bean> </beans> HikariCP Pool Size (Maximum Pool Size) Here’s a detailed guide on why and how to avoid setting a higher connection pool size: For example, with 2400 TPM (which is 40 TPS) and each query taking about 1 second, you may need around 40 connections, with an additional buffer of 10-20% to handle spikes, giving a total pool size of around 44-48 connections. Caution: Avoiding an excessively high connection pool size in HikariCP (or any connection pooling mechanism) is critical for optimizing resource usage and ensuring stable application performance. A higher-than-necessary pool size can lead to resource contention, database overload, and system instability. 1. Right-Size the Pool Use the formula below to calculate the optimal pool size: CSS Optimal Pool Size = (Core Threads) × (1 + (Wait Time / Service Time)) Optimal Pool Size = (Core Threads) × (1 + (Service Time / Wait Time)) Core Threads: Number of threads available for executing queries.Wait Time: Time the application can afford to wait for a connection.Service Time: Average time for a query to execute. 2. Connection Timeout Set the connectionTimeout to a value less than your SLA, such as 500-700 milliseconds, to ensure that connections are not held up for too long if they are not available. 3. Idle Timeout Configure idleTimeout to a lower value, like 30,000 ms (30 seconds), so that idle connections are quickly released, avoiding resource waste. 4. Max Lifetime Set the maxLifetime slightly shorter than the database’s connection timeout (e.g., 30 minutes) to avoid connections being closed abruptly by the database. 5. Connection Validation Use validationQuery or enable validationTimeout to ensure connections are always valid, keeping the latency minimal. 6. Database Connection Utilization Ensure that queries are optimized for performance on the database side.Monitor database resources (CPU, memory, and indexes) to see if adjustments can be made for better utilization. These settings should improve how connections are used and help achieve your target response time of under 1 second. If you’re still experiencing issues, consider analyzing query performance and optimizing database operations. Step 4. Configure Spring in Mule Application Global Elements Add this Spring configuration in global.xml and refer to the config-ref: XML <mule xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:db="http://www.mulesoft.org/schema/mule/db" xmlns:spring="http://www.mulesoft.org/schema/mule/spring" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd http://www.mulesoft.org/schema/mule/spring http://www.mulesoft.org/schema/mule/spring/current/mule-spring.xsd"> <!--Spring beans--> <spring:config name="Spring_Config" doc:name="Spring Config" doc:id="5550c804-20cf-40c0-9331-d3ee45d7444f" files="spring-beans.xml" /> <!--DB Config.. Reference the datasource created in Spring beans--> <db:config name="Database_Config" doc:name="Database Config" doc:id="f18a2e4d-0f43-4adc-86f3-f76a42ecc3c9" > <db:data-source-connection dataSourceRef="dataSource" /> </db:config> <!-- Database Configuration --> </mule> Step 5: Use the Spring Bean in Mule Flows The datasource bean is now available in the MuleSoft application and can be referenced in the db module configuration. Step 6: Verify Configuration Run the Mule application to ensure the Spring context is loaded correctly.Test database interactions to validate the connection pooling behavior. Step 7: Simulate Load Tests for HikariCP Pool Size Configuration Test Scenario API Use Case: Mule 4 API handling database queries with HikariCP connection pooling.Transaction Load: 3000 Transactions Per Minute (TPM).Concurrency: 50 concurrent users.Query Execution Time: 200ms per query.Connection Pool Configuration: Max Pool Size: 240Min Pool Size: 20Idle Timeout: 30 secondsMax Lifetime: 30 minutesLeak Detection Threshold: 2 seconds Conclusion Integrating HikariCP into your MuleSoft applications unlocks a new level of performance, reliability, and scalability for database interactions. By utilizing HikariCP’s efficient connection pooling and combining it with MuleSoft’s robust integration capabilities, you can confidently handle high-traffic workloads and demanding SLAs. Whether you're building APIs, processing real-time data, or managing complex integrations, HikariCP ensures optimal use of resources, reduced latency, and seamless scalability. With proper configuration and thoughtful integration, you can transform your MuleSoft applications into high-performance engines ready for modern enterprise challenges.

AI/ML workflows excel on structured, reliable data pipelines. To streamline these processes, DBT and Snowpark offer complementary capabilities: DBT is for modular SQL transformations, and Snowpark is for programmatic Python-driven feature engineering. Here are some key benefits of using DBT, Snowpark, and Snowflake together: Simplifies SQL-based ETL with DBT’s modularity and tests.Handles complex computations with Snowpark’s Python UDFs.Leverages Snowflake’s high-performance engine for large-scale data processing. Here’s a step-by-step guide to installing, configuring, and integrating DBT and Snowpark into your workflows. Step 1: Install DBT In Shell, you can use Python’s pip command for installing packages. Assuming Python is already installed and added to your PATH, follow these steps: Shell # Set up a Python virtual environment (recommended): python3 -m venv dbt_env source dbt_env/bin/activate # Install DBT and the Snowflake adapter: pip install dbt-snowflake # Verify DBT installation dbt --version Step 2: Install Snowpark Shell # Install Snowpark for Python pip install snowflake-snowpark-python # Install additional libraries for data manipulation pip install pandas numpy # Verify Snowpark installation python -c "from snowflake.snowpark import Session; print('successful Snowpark installation')" Step 3: Configuring DBT for Snowflake DBT requires a profiles.yml file to define connection settings for Snowflake. Locate the DBT Profiles Directory By default, DBT expects the profiles.yml file in the ~/.dbt/ directory. Create the directory if it doesn’t exist: Shell mkdir -p ~/.dbt Create the profiles.yml File Define your Snowflake credentials in the following format: YAML my_project: outputs: dev: type: snowflake account: your_account_identifier user: your_username password: your_password role: your_role database: your_database warehouse: your_warehouse schema: your_schema target: dev Replace placeholders like your_account_identifier with your Snowflake account details. Test the Connection Run the following command to validate your configuration: Shell dbt debug If the setup is correct, you’ll see a success message confirming the connection. Step 4: Setting Up Snowpark Ensure Snowflake Permissions Before using Snowpark, ensure your Snowflake user has the following permissions: Access to the warehouse and schema.Ability to create and register UDFs (User-Defined Functions). Create a Snowpark Session Set up a Snowpark session using the same credentials from profiles.yml: Python from snowflake.snowpark import Session def create_session(): connection_params = { "account": "your_account_identifier", "user": "your_username", "password": "your_password", "role": "your_role", "database": "your_database", "warehouse": "your_warehouse", "schema": "your_schema", } return Session.builder.configs(connection_params).create() session = create_session() print("Snowpark session created successfully") Register a Sample UDF Here’s an example of registering a simple Snowpark UDF for text processing: Python def clean_text(input_text): return input_text.strip().lower() session.udf.register( func=clean_text, name="clean_text_udf", input_types=["string"], return_type="string", is_permanent=True ) print("UDF registered successfully") Step 5: Integrating DBT With Snowpark You have a DBT model named raw_table that contains raw data. raw_table DBT Model Definition SQL -- models/raw_table.sql SELECT * FROM my_database.my_schema.source_table Use Snowpark UDFs in DBT Models Once you’ve registered a UDF in Snowflake using Snowpark, you can call it directly from your DBT models. SQL -- models/processed_data.sql WITH raw_data AS ( SELECT id, text_column FROM {{ ref('raw_table') } ), cleaned_data AS ( SELECT id, clean_text_udf(text_column) AS cleaned_text FROM raw_data ) SELECT * FROM cleaned_data; Run DBT Models Execute your DBT models to apply the transformation: Shell dbt run --select processed_data Step 6: Advanced AI/ML Use Case For AI/ML workflows, Snowpark can handle tasks like feature engineering directly in Snowflake. Here’s an example of calculating text embeddings: Create an Embedding UDF Using Python and a pre-trained model, you can generate text embeddings: Python from transformers import pipeline def generate_embeddings(text): model = pipeline("feature-extraction", model="bert-base-uncased") return model(text)[0] session.udf.register( func=generate_embeddings, name="generate_embeddings_udf", input_types=["string"], return_type="array", is_permanent=True ) Integrate UDF in DBT Call the embedding UDF in a DBT model to create features for ML: SQL -- models/embedding_data.sql WITH raw_text AS ( SELECT id, text_column FROM {{ ref('raw_table') } ), embedded_text AS ( SELECT id, generate_embeddings_udf(text_column) AS embeddings FROM raw_text ) SELECT * FROM embedded_text; Best Practices Use DBT for reusable transformations: Break down complex SQL logic into reusable models.Optimize Snowpark UDFs: Write lightweight, efficient UDFs to minimize resource usage.Test Your Data: Leverage DBT’s testing framework for data quality.Version Control Everything: Track changes in DBT models and Snowpark scripts for traceability. Conclusion By combining DBT’s SQL-based data transformations with Snowpark’s advanced programming capabilities, you can build AI/ML pipelines that are both scalable and efficient. This setup allows teams to collaborate effectively while leveraging Snowflake’s computational power to process large datasets. Whether you’re cleaning data, engineering features, or preparing datasets for ML models, the DBT-Snowpark integration provides a seamless workflow to unlock your data’s full potential.

As we reach the end of 2024, JavaScript continues to dominate as the go-to programming language for web development, owing to its versatility and community-driven evolution. The latest ECMAScript updates introduce several powerful features aimed at improving developer productivity, code readability, and overall efficiency. 1. Pipeline Operator (|>) The pipeline operator is one of the most anticipated features of ES2024. Borrowed from functional programming paradigms, this operator improves the readability and maintainability of complex function chains by enabling a linear, step-by-step flow of data through multiple functions. How It Works Traditionally, chaining multiple functions requires nested calls, which can quickly become hard to read: JavaScript const result = uppercase(exclaim(addGreeting("Hello"))); With the pipeline operator, the same logic can be written in a cleaner and more readable way: JavaScript const result = uppercase(exclaim(addGreeting("Hello"))); Here, % acts as a placeholder for the value passed from the previous operation. This simple syntax improves code readability, especially in projects requiring complex data transformations. Use Cases The pipeline operator is particularly useful for: Functional Programming: Chaining small, reusable functions in a clear sequence.Data Processing: Applying multiple transformations to datasets in a readable manner.Simplifying Async Workflows: Integrating with await to make asynchronous pipelines intuitive (pending further committee discussions on compatibility). 2. Regular Expression Enhancements (v Flag) ES2024 introduces significant improvements to regular expressions with the addition of the v flag. This enhancement provides powerful new operators (intersection (&&), difference (--), and union (||)) that simplify complex pattern matching. Key Features Intersection (&&) Matches characters that are common to two character sets. For example: JavaScript let regex = /[[a-z]&&[^aeiou]]/v; console.log("crypt".match(regex)); // Matches consonants only: ['c', 'r', 'p', 't'] Difference (--) Excludes specific characters from a set: JavaScript let regex = /[\p{Decimal_Number}--[0-9]]/v; console.log("123٤٥٦".match(regex)); // Matches non-ASCII numbers: ['٤', '٥', '٦'] Union (||) Combines multiple sets, allowing broader matches. Practical Applications These operators simplify patterns for advanced text processing tasks, such as: Filtering non-ASCII characters or special symbols in multilingual datasets.Creating fine-tuned matches for validation tasks (e.g., email addresses, custom identifiers).Extracting domain-specific patterns like mathematical symbols or emojis. 3. Temporal API The Temporal API finally provides a modern, robust replacement for the outdated Date object, addressing long-standing issues with time zones, date manipulations, and internationalization. Why Temporal? The existing Date object has numerous flaws, including: Poor handling of time zones.Complex workarounds for date arithmetic.Limited support for non-Gregorian calendars. Temporal offers a more comprehensive and intuitive API for working with dates, times, and durations. Example: JavaScript const date = Temporal.PlainDate.from("2024-11-21"); const futureDate = date.add({ days: 10 }); console.log(futureDate.toString()); // Outputs: 2024-12-01 Core Features Precise Time Zones: Built-in handling of time zone offsets and daylight saving changes.Immutable Objects: Prevents accidental modifications, making code safer.Duration Arithmetic: Simplifies operations like adding or subtracting time units. Use Cases The Temporal API is ideal for: Scheduling Systems: Handling recurring events with time zone precision.Global Applications: Supporting international users with different calendars.Financial Applications: Accurate interest and payment calculations across time periods. 4. Object.groupBy() Grouping array elements based on specific criteria is a common requirement, and ES2024’s Object.groupBy() method simplifies this process significantly. How It Works Previously, developers had to write custom functions or rely on libraries like Lodash for grouping operations. With Object.groupBy(), grouping becomes straightforward: JavaScript const data = [ { type: "fruit", name: "apple" }, { type: "vegetable", name: "carrot" }, { type: "fruit", name: "banana" }, ]; const grouped = Object.groupBy(data, item => item.type); console.log(grouped); // Output: // { // fruit: [{ type: "fruit", name: "apple" }, { type: "fruit", name: "banana" }], // vegetable: [{ type: "vegetable", name: "carrot" }] // } Advantages Simplicity: Eliminates the need for custom grouping logic.Readability: Provides a declarative approach to data organization.Performance: Optimized for modern JavaScript engines. Applications Categorizing datasets for dashboards or analytics tools.Organizing user input based on metadata, such as tags or roles.Simplifying the preparation of reports from raw data. 5. Records and Tuples The records and tuples proposal introduces immutable data structures to JavaScript, ensuring data safety and reducing unintended side effects. What Are They? Records: Immutable key-value pairs, similar to objects.Tuples: Immutable, ordered lists, similar to arrays. Example: JavaScript const record = #{ name: "Alice", age: 25 }; const tuple = #[1, 2, 3]; console.log(record.name); // Output: "Alice" console.log(tuple[1]); // Output: 2 Key Benefits Immutability: Helps prevent bugs caused by accidental data mutations.Simplified Equality Checks: Records and Tuples are deeply compared by value, unlike objects and arrays. Use Cases Storing configuration data that must remain unchanged.Implementing functional programming techniques.Creating reliable data snapshots in state management systems like Redux. Conclusion With ES2024, JavaScript solidifies its position as a cutting-edge programming language that evolves to meet the demands of modern development. The pipeline operator, regex enhancements, Temporal API, Object.groupBy(), and immutable data structures like records and tuples are poised to streamline workflows and solve long-standing challenges. As these features gain adoption across browsers and Node.js environments, developers should explore and integrate them to write cleaner, more efficient, and future-proof code.

If you are already using DuckDB, this guide will help you with some optimization techniques that can improve your application's performance. If you are new to DuckDB, don't fret — you'll still learn something new. I will share some of the practical tips that helped me optimize my applications. Let's dive in! Why DuckDB? Before we jump into optimization techniques, let's quickly discuss what makes DuckDB stand out. In the official DuckDB documentation, many benefits are listed. Give it a read. Loading Data One thing to remember about data loading in DuckDB is that the file format you choose makes a huge difference. Here's what I've learned: SQL -- Here's how I usually load my Parquet files SELECT * FROM read_parquet('my_data.parquet'); -- And here's a neat trick for CSV files CREATE TABLE my_table AS SELECT * FROM read_csv_auto('my_data.csv'); Tip: If you're working with CSV files, consider converting them to Parquet. Parquet files are compressed, columnar, and super fast to query. Chunking: Because Size Matters! I've successfully processed datasets in chunks, especially the larger ones. It's not only efficient, but also can help you debug any issues smoothly. Python import duckdb import pandas as pd def process_big_data(file_path, chunk_size=100000): # Let's break this elephant into bite-sized pieces conn = duckdb.connect(':memory:') print("Ready to tackle this big dataset!") processed_count = 0 while True: # Grab a chunk chunk = conn.execute(f""" SELECT * FROM read_csv_auto('{file_path}') LIMIT {chunk_size} OFFSET {processed_count} """).fetchdf() if len(chunk) == 0: break # Implement your logic here process_chunk(chunk) processed_count += len(chunk) print(f"Processed {processed_count:,} rows... Keep going!") I like to think of this as eating a pizza — you wouldn't try to stuff the whole thing in your mouth at once, right? The same goes for data processing. Query Optimization I've used some queries that would make any database. I learned some of the best practices the hard way (well, hard on the database, too). Here are some tips: 1. Use EXPLAIN ANALYZE to See What's Happening Under the Hood This will show you exactly how DuckDB is processing your query. This should inform you how to further tune your query. SQL EXPLAIN ANALYZE SELECT category, COUNT(*) as count FROM sales WHERE date >= '2024-01-01' GROUP BY category; 2. Be Specific With Columns It's like packing for a weekend trip — do you really need to bring your entire wardrobe? SQL -- Good: Only fetching what we need SELECT user_id, SUM(amount) as total_spent FROM purchases WHERE category = 'books' GROUP BY user_id; -- Not great: Why fetch all columns when we only need two? SELECT * FROM purchases WHERE category = 'books'; 3. Smart Joins Make Happy Databases It's more like organizing a party — you wouldn't invite everyone in town and then figure out who knows each other, right? SQL - Good: Filtering before the join SELECT u.name, o.order_date FROM users u JOIN orders o ON u.id = o.user_id WHERE u.country = 'Canada' AND o.status = 'completed'; -- Not optimal: Joining everything first SELECT u.name, o.order_date FROM (SELECT * FROM users) u JOIN (SELECT * FROM orders) o ON u.id = o.user_id WHERE u.country = 'Canada' AND o.status = 'completed'; 4. Window Functions Done Right It's like keeping a running score in a game — you update as you go, not by recounting all points for each play. SQL - Good: Efficient window function usage SELECT product_name, sales_amount, SUM(sales_amount) OVER (PARTITION BY category ORDER BY sale_date) as running_total FROM sales WHERE category = 'electronics'; -- Less efficient: Using subqueries instead SELECT s1.product_name, s1.sales_amount, (SELECT SUM(sales_amount) FROM sales s2 WHERE s2.category = s1.category AND s2.sale_date <= s1.sale_date) as running_total FROM sales s1 WHERE category = 'electronics'; Memory Management Here's another thing that I learned the hard way: always set memory limits. Here's how I keep things under control: Set memory_limit to 50-70% of available system RAMSet max_memory to about half of memory_limitMonitor and adjust based on your workload First, let's understand how DuckDB uses memory: SQL -- Check current memory settings SELECT * FROM duckdb_settings() WHERE name LIKE '%memory%'; -- View current memory usage SELECT * FROM pragma_database_size(); Basic Memory Settings Think of these settings as setting a budget for your shopping: memory_limit is like your total monthly budgetmax_memory is like setting a limit for each shopping triptemp_directory is like having a storage unit when your closet gets full SQL -- Set overall memory limit SET memory_limit='4GB'; -- Set maximum memory per query SET temp_directory='/path/to/disk'; -- For spilling to disk SET max_memory='2GB'; -- Per-query limit Monitoring Memory Usage SQL -- Enable progress bar to monitor operations SET enable_progress_bar=true; -- Enable detailed profiling SET enable_profiling=true; PRAGMA enable_profiling; -- After running your query, check the profile PRAGMA profile; Memory-Related Warning Signs Watch out for these signs of memory pressure: Slow query performanceSystem becoming unresponsiveQuery failures with "out of memory" errorsExcessive disk activity (spilling to disk) Clean Up Regularly Drop temporary tables and vacuum when needed: SQL -- Clean up temporary objects DROP TABLE IF EXISTS temp_results; VACUUM; Conclusion Always start with the basics, measure everything, and optimize where it matters most. Here's a quick checklist I use: Is my data in the right format? (Parquet is usually the answer)Am I processing data in chunks when dealing with large datasets?Are my queries as specific as possible?Have I set reasonable memory limits?

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Observability and Performance: The Precipice of Building Highly Performant Software Systems. "Quality is not an act, it's a habit," said Aristotle, a principle that rings true in the software world as well. Specifically for developers, this means delivering user satisfaction is not a one-time effort but an ongoing commitment. To achieve this commitment, engineering teams need to have reliability goals that clearly define the baseline performance that users can expect. This is precisely where service-level objectives (SLOs) come into picture. Simply put, SLOs are reliability goals for products to achieve in order to keep users happy. They serve as the quantifiable bridge between abstract quality goals and the day-to-day operational decisions that DevOps teams must make. Because of this very importance, it is critical to define them effectively for your service. In this article, we will go through a step-by-step approach to define SLOs with an example, followed by some challenges with SLOs. Steps to Define Service-Level Objectives Like any other process, defining SLOs may seem overwhelming at first, but by following some simple steps, you can create effective objectives. It's important to remember that SLOs are not set-and-forget metrics. Instead, they are part of an iterative process that evolves as you gain more insight into your system. So even if your initial SLOs aren't perfect, it's okay — they can and should be refined over time. Figure 1. Steps to define SLOs Step 1: Choose Critical User Journeys A critical user journey refers to the sequence of interactions a user takes to achieve a specific goal within a system or a service. Ensuring the reliability of these journeys is important because it directly impacts the customer experience. Some ways to identify critical user journeys can be through evaluating revenue/business impact when a certain workflow fails and identifying frequent flows through user analytics. For example, consider a service that creates virtual machines (VMs). Some of the actions users can perform on this service are browsing through the available VM shapes, choosing a region to create the VM in, and launching the VM. If the development team were to order them by business impact, the ranking would be: Launching the VM because this has a direct revenue impact. If users cannot launch a VM, then the core functionality of the service has failed, affecting customer satisfaction and revenue directly.Choosing a region to create the VM. While users can still create a VM in a different region, it may lead to a degraded experience if they have a regional preference. This choice can affect performance and compliance.Browsing through the VM catalog. Although this is important for decision making, it has a lower direct impact on the business because users can change the VM shape later. Step 2: Determine Service-Level Indicators That Can Track User Journeys Now that the user journeys are defined, the next step is to measure them effectively. Service-level indicators (SLIs) are the metrics that developers use to quantify system performance and reliability. For engineering teams, SLIs serve a dual purpose: They provide actionable data to detect degradation, guide architectural decisions, and validate infrastructure changes. They also form the foundation for meaningful SLOs by providing the quantitative measurements needed to set and track reliability targets. For instance, when launching a VM, some of the SLIs can be availability and latency. Availability: Out of the X requests to launch a VM, how many succeeded? A simple formula to calculate this is: If there were 1,000 requests and 998 requests out of them succeeded, then the availability is = 99.8%. Latency: Out of the total number of requests to launch a VM, what time did the 50th, 95th, or 99th percentile of requests take to launch the VM? The percentiles here are just examples and can vary depending on the specific use case or service-level expectations. In a scenario with 1,000 requests where 900 requests were completed in 5 seconds and the remaining 100 took 10 seconds, the 95th percentile latency would be = 10 seconds.While averages can also be used to calculate latencies, percentiles are typically recommended because they account for tail latencies, offering a more accurate representation of the user experience. Step 3: Identify Target Numbers for SLOs Simply put, SLOs are the target numbers we want our SLIs to achieve in a specific time window. For the VM scenario, the SLOs can be: The availability of the service should be greater than 99% over a 30-day rolling window.The 95th percentile latency for launching the VMs should not exceed eight seconds. When setting these targets, some things to keep in mind are: Using historical data. If you need to set SLOs based on a 30-day rolling period, gather data from multiple 30-day windows to define the targets. If you lack this historical data, start with a more manageable goal, such as aiming for 99% availability each day, and adjust it over time as you gather more information. Remember, SLOs are not set in stone; they should continuously evolve to reflect the changing needs of your service and customers. Considering dependency SLOs. Services typically rely on other services and infrastructure components, such as databases and load balancers. For instance, if your service depends on a SQL database with an availability SLO of 99.9%, then your service's SLO cannot exceed 99.9%. This is because the maximum availability is constrained by the performance of its underlying dependencies, which cannot guarantee higher reliability. Challenges of SLOs It might be intriguing to set the SLO as 100%, but this is impossible. A 100% availability, for instance, means that there is no room for important activities like shipping features, patching, or testing, which is not realistic. Defining SLOs requires collaboration across multiple teams, including engineering, product, operations, QA, and leadership. Ensuring that all stakeholders are aligned and agree on the targets is essential for the SLO to be successful and actionable. Step 4: Account for Error Budget An error budget is the measure of downtime a system can afford without upsetting customers or breaching contractual obligations. Below is one way of looking at it: If the error budget is nearly depleted, the engineering team should focus on improving reliability and reducing incidents rather than releasing new features.If there's plenty of error budget left, the engineering team can afford to prioritize shipping new features as the system is performing well within its reliability targets. There are two common approaches to measuring the error budget: time based and event based. Let's explore how the statement, "The availability of the service should be greater than 99% over a 30-day rolling window," applies to each. Time-Based Measurement In a time-based error budget, the statement above translates to the service being allowed to be down for 43 minutes and 50 seconds in a month, or 7 hours and 14 minutes in a year. Here's how to calculate it: Determine the number of data points. Start by determining the number of time units (data points) within the SLO time window. For instance, if the base time unit is 1 minute and the SLO window is 30 days: Calculate the error budget. Next, calculate how many data points can "fail" (i.e., downtime). The error budget is the percentage of allowable failure. Convert this to time: This means the system can experience 7 hours and 14 minutes of downtime in a 30-day window. Last but not least, the remaining error budget is the difference between the total possible downtime and the downtime already used. Event-Based Measurement For event-based measurement, the error budget is measured in terms of percentages. The aforementioned statement translates to a 1% error budget in a 30-day rolling window. Let's say there are 43,200 data points in that 30-day window, and 100 of them are bad. You can calculate how much of the error budget has been consumed using this formula: Now, to find out how much error budget remains, subtract this from the total allowed error budget (1%): Thus, the service can still tolerate 0.77% more bad data points. Advantages of Error Budget Error budgets can be utilized to set up automated monitors and alerts that notify development teams when the budget is at risk of depletion. These alerts enable them to recognize when a greater caution is required while deploying changes to production. Teams often face ambiguity when it comes to prioritizing features vs. operations. Error budget can be one way to address this challenge. By providing clear, data-driven metrics, engineering teams are able to prioritize reliability tasks over new features when necessary. The error budget is among the well-established strategies to improve accountability and maturity within the engineering teams. Cautions to Take With Error Budgets When there is extra budget available, developers should actively look into using it. This is a prime opportunity to deepen the understanding of the service by experimenting with techniques like chaos engineering. Engineering teams can observe how the service responds and uncover hidden dependencies that may not be apparent during normal operations. Last but not least, developers must monitor error budget depletion closely as unexpected incidents can rapidly exhaust it. Conclusion Service-level objectives represent a journey rather than a destination in reliability engineering. While they provide important metrics for measuring service reliability, their true value lies in creating a culture of reliability within organizations. Rather than pursuing perfection, teams should embrace SLOs as tools that evolve alongside their services. Looking ahead, the integration of AI and machine learning promises to transform SLOs from reactive measurements into predictive instruments, enabling organizations to anticipate and prevent failures before they impact users. Additional resources: Implementing Service Level Objectives, Alex Hidalgo, 2020 "Service Level Objects," Chris Jones et al., 2017 "Implementing SLOs," Steven Thurgood et al., 2018 Uptime/downtime calculator This is an excerpt from DZone's 2024 Trend Report, Observability and Performance: The Precipice of Building Highly Performant Software Systems.Read the Free Report

By Siri Varma Vegiraju CORE

Look, I'll be honest — when my team first started using AI coding assistants last year, I was skeptical — really skeptical. After 15 years of writing code, I didn't believe a language model could meaningfully help with real development work. Six months later, I had to eat my words. Our team's velocity increased by roughly 40%, and our code quality metrics actually improved. But here's the thing - it's not as simple as "AI makes coding easier." The reality is more nuanced, more interesting, and frankly, more useful than the marketing hype suggests. The Reality on the Ground Let me share something that happened last week. I was debugging a nasty memory leak in our Node.js backend. Traditionally, this would have meant hours of combing through code, adding console logs, and maybe buying more coffee. Instead, I fed the relevant code and memory profiles into our AI assistant and asked it to analyze the patterns. Within minutes, it spotted something I'd missed — we were accidentally holding onto WebSocket connections in a closure. This was a simple mistake, tricky to spot. The AI didn't magically fix it, but it pointed me in the right direction much faster than traditional debugging would have. What Actually Works Based on my team's experience over the past year: 1. Code Reviews The biggest surprise was how useful AI became in code reviews. Example from our codebase: Original code: Python def process_user_data(data): if data['status'] == 'active': user = User.objects.get(id=data['user_id']) if user.subscription: if user.subscription.status == 'valid': return handle_active_user(user) else: return handle_inactive_subscription(user) return handle_inactive_user() Our AI assistant flagged this and suggested the following: Refactored version: Python def process_user_data(data): if data['status'] != 'active': return handle_inactive_user() user = User.objects.get(id=data['user_id']) if not user.subscription: return handle_inactive_user() return (handle_active_user(user) if user.subscription.status == 'valid' else handle_inactive_subscription(user)) It is not revolutionary but consistently catches these readability issues faster than human reviewers. 2. Test Generation This was another surprise winner. Here's a real example: Original function: Python def calculate_order_total(items, discount_code=None): subtotal = sum(item.price * item.quantity for item in items) if discount_code: discount = get_discount_amount(discount_code, subtotal) return subtotal - discount return subtotal The AI-generated these test cases: Python def test_calculate_order_total(): items = [ Item(price=10.0, quantity=2), Item(price=15.0, quantity=1) ] assert calculate_order_total(items) == 35.0 def test_calculate_order_total_with_discount(): items = [Item(price=100.0, quantity=1)] assert calculate_order_total(items, 'SAVE20') == 80.0 def test_calculate_order_total_empty(): assert calculate_order_total([]) == 0 def test_calculate_order_total_invalid_discount(): items = [Item(price=10.0, quantity=1)] assert calculate_order_total(items, 'INVALID') == 10.0 It caught edge cases I probably would have missed initially. What Doesn't Work 1. Complex Architectural Decisions Last month, we needed to redesign our authentication system to handle multi-tenant SSO. The AI suggestions were technically correct but missed crucial real-world considerations about session management across different client types. 2. Context-Heavy Debugging When a bug involves multiple services, complex state, or race conditions, AI tools still struggle. They can help analyze individual components but often miss the bigger picture. Real-World Impact on Development Workflow Here's how our development workflow has actually changed: Before AI Integration Plain Text 1. Write code (2 hours) 2. Write tests (1 hour) 3. Debug issues (2 hours) 4. Code review (1 hour) Total: ~6 hours per feature After AI Integration Plain Text 1. Write code with AI assistance (1 hour) 2. AI generates test cases, developer adjusts (30 mins) 3. Debug with AI analysis (1 hour) 4. AI-assisted code review (30 mins) Total: ~3 hours per feature But here's the catch - this only works because we learned how to use AI tools effectively. The first month was actually slower as we figured out the right workflows. Best Practices We've Learned 1. Prompt Engineering for Developers Instead of: Plain Text "Write a function to process user data" We do: Plain Text "Write a Python function that: - Takes a user_id and data dictionary - Validates required fields: name, email, age - Handles missing fields with default values - Raises ValidationError for invalid data - Returns a processed user object Use type hints and include error handling." The difference in output quality is dramatic. 2. Code Review Strategy We now do: First pass: AI review for style, potential bugs, and obvious issuesSecond pass: Human review for business logic and architectural concernsFinal pass: AI check for security vulnerabilities 3. Documentation Generation This has been a game-changer. Example: Python # Original function with minimal docs def process_payment(amount, user_id, method): # ... implementation ... AI expanded this to: Python def process_payment( amount: Decimal, user_id: str, method: PaymentMethod ) -> PaymentResult: """Process a payment transaction for a user. Args: amount: Transaction amount in user's local currency. user_id: Unique identifier for the user. method: Payment method object containing card/bank details. Returns: PaymentResult object containing transaction ID and status. Raises: InsufficientFundsError: If payment method has insufficient funds. InvalidPaymentMethodError: If payment method is expired/invalid. PaymentProcessingError: If payment gateway encounters an error. Example: >>> result = process_payment( ... amount=Decimal('99.99'), ... user_id='usr_123', ... method=PaymentMethod(type='credit_card', token='tok_xyz') ... ) >>> print(result.transaction_id) 'tx_abc123' """ Security Considerations One area where we've had to be extremely careful is security. Some lessons learned: 1. Never Let AI Generate Security-Critical Code Example of what not to do: Python # DON'T: Let AI generate authentication logic def verify_password(plain_text, hashed): return hashlib.md5(plain_text.encode()).hexdigest() == hashed 2. Always Review Generated SQL We've seen AI suggest vulnerable queries: SQL -- DON'T: Raw string formatting f"SELECT * FROM users WHERE id = '{user_id}'" -- DO: Parameterized queries "SELECT * FROM users WHERE id = %s", (user_id,) Looking Forward Based on current trends and my experience, here's what's actually changing: 1. IDE Integration Is Getting Serious The latest AI-powered IDEs don't just suggest code - they understand entire codebases. Last week, our IDE flagged a potential race condition in an async function by analyzing how it's called across different services. 2. Specialized Models Are Coming We're seeing AI models trained specifically for certain frameworks or languages. The TypeScript-specific suggestions we're getting now are notably better than generic code generation. 3. Testing Is Being Transformed AI is getting better at generating edge cases and stress tests that humans might miss. Our test coverage has actually increased since adopting AI tools. Conclusion Look, AI isn't replacing developers anytime soon. What it is doing is making us more efficient, helping catch bugs earlier, and handling the boring parts of coding. The key is understanding its limitations and using it as a tool, not a replacement for human judgment. The developers who'll thrive in this new environment aren't the ones who can write the most code - they're the ones who can effectively collaborate with AI tools while maintaining a clear vision of what they're building and why. And yes, I used AI to help write parts of this article. That's the point — it's a tool, and I'm not ashamed to use it effectively.

NodeJS is a very popular platform for building backend services and creating API endpoints. Several large companies use NodeJS in their microservices tech stack, which makes it a very useful platform to learn and know, similar to other popular languages like Java and Python. ExpressJS is a leading framework used for building APIs, and TypeScript provides necessary strict typing support, which is very valuable in large enterprise application development. TypeScript and ExpressJS combined together allow the development of robust distributed backend systems. Securing access to such a system is very critical. The NodeJS platform offers several options for securing APIs, such as JWT (JSON Web Token), OAuth2, Session-based authentication, and more. JWT has seen a rise in adoption due to several key characteristics when it comes to securing the APIs. Some of the noteworthy benefits of using JWT to secure APIs are noted below: Stateless authentication: JWT tokens carry all necessary information for authentication with them and don't need any server-side storage.Compact and efficient: JWT tokens are small, allowing for easy transmission over the network. They can be easily sent in HTTP headers.CORS-support: As JWT tokens are stateless, they make it super easy to implement cross-browser support. This makes them ideal for Single Page Applications (SPAs) and microservices architectures.Standardized format: JWT tokens follow RFC 7519 - JWT specification, which makes them ideal for cross-platform interoperability. In this tutorial, you will be building NodeJS-based microservices from scratch using Express and TypeScript in the beginning. The tutorial implements a library book management system where a user can view as well as edit a catalog of books. Later, you will be securing the endpoints of this microservice using JWT. The full code for the tutorial is available on this GitHub link. However, I encourage you to follow along for deeper insights and understanding. Prerequisites To follow along in the tutorial, ensure the below prerequisites are met. Understanding of JavaScript. TypeScript familiarity is a great bonus to have. Understanding of REST API operations, such as GET, POST, PUT, and DELETE.NodeJS and NPM installed on the machine. This can be verified using node -v and npm -v.An editor of choice. Visual Studio Code was used in the development of this tutorial and is a good choice. Initiating the New NodeJS App Create a new folder on your local machine and initiate a new NodeJS application using the commands below: GitHub Flavored Markdown mkdir ts-node-jwt cd ts-node-jwt npm init -y The NodeJS you will build uses TypeScript and ExpressJS. Install necessary dependencies using the npm commands below: GitHub Flavored Markdown npm install typescript ts-node-dev @types/node --save-dev npm install express dotenv npm install @types/express --save-dev The next step is to initiate and define TypeScript configuration. Use the command below to create a new TypeScript configuration: GitHub Flavored Markdown npx tsc --init At this point, open the project folder in your editor and locate the freshly created tsconfig.json file and update its content as per below: JSON { "compilerOptions": { "target": "ES6", "module": "commonjs", "strict": true, "esModuleInterop": true, "forceConsistentCasingInFileNames": true, "skipLibCheck": true, "outDir": "./dist" }, "include": ["src"], "exclude": ["node_modules"] } Creating the Basic App With No Authentication Create a folder named src inside the project root, and inside this src directory, create a file name server.ts. This file will contain basic server boot-up code. TypeScript /*Path of the file: project-root/src/server.ts*/ import express from 'express'; import dotenv from 'dotenv'; import router from './routes'; dotenv.config(); const app = express(); const PORT = process.env.PORT || 3000; app.use(express.json()); app.use('/api', router); app.listen(PORT, () => { console.log(`Server is running on http://localhost:${PORT}`); }); Create a directory named routes under src and add file index.ts to it. This file will hold routing details and handling of routes for APIs needed to implement the catalog system that you are building. TypeScript /*Path of the file: project-root/src/routes/index.ts*/ import { Router } from 'express'; import { getAllBooks, getBookById, addNewBook, removeBook } from '../controllers/bookController'; const router = Router(); router.get('/books', getAllBooks); router.get('/books/:id', getBookById); router.post('/books', addNewBook); router.delete('/books/:id', removeBook); export default router; Next, you should create a book controller. This controller will hold code for handling, receiving, and responding to actual API calls. Create controllers directory under src. Add a file named bookController.ts under src/controllers directory. Add the code below to this file. This controller code receives each individual API call, parses its request when needed, then interacts with the service layer (which you will build in the next steps), and responds to the user. TypeScript /*Path of the file: project-root/src/controllers/userController.ts*/ import { Request, Response } from 'express'; import { getBooks, findBookById, addBook, deleteBook } from '../services/bookService'; export const getAllBooks = (req: Request, res: Response): void => { const books = getBooks(); res.json(books); }; export const getBookById = (req: Request, res: Response): void => { const bookId = parseInt(req.params.id); if (isNaN(bookId)) { res.status(400).json({ message: 'Invalid book ID' }); return; } const book = findBookById(bookId); if (!book) { res.status(404).json({ message: 'Book not found' }); return; } res.json(book); }; export const addNewBook = (req: Request, res: Response): void => { const { title, author, publishedYear } = req.body; if (!title || !author || !publishedYear) { res.status(400).json({ message: 'Missing required fields' }); return; } const newBook = { id: Date.now(), title, author, publishedYear }; addBook(newBook); res.status(201).json(newBook); }; export const removeBook = (req: Request, res: Response): void => { const bookId = parseInt(req.params.id); if (isNaN(bookId)) { res.status(400).json({ message: 'Invalid book ID' }); return; } const book = findBookById(bookId); if (!book) { res.status(404).json({ message: 'Book not found' }); return; } deleteBook(bookId); res.status(200).json({ message: 'Book deleted successfully' }); }; The controller interacts with the book service to perform reads and writes on the book database. Create a JSON file as per below with dummy books, which will act as the database. JSON /*Path of the file: project-root/src/data/books.json*/ [ { "id": 1, "title": "To Kill a Mockingbird", "author": "Harper Lee", "publishedYear": 1960 }, { "id": 2, "title": "1984", "author": "George Orwell", "publishedYear": 1949 }, { "id": 3, "title": "Pride and Prejudice", "author": "Jane Austen", "publishedYear": 1813 } ] Read this book's details in the service file and provide methods for updating the books as well. This code implements an in-memory book database. Add a directory services under src and add file bookService.ts with the code below. TypeScript /*Path of the file: project-root/src/services/bookService.ts*/ import fs from 'fs'; import path from 'path'; interface Book { id: number; title: string; author: string; publishedYear: number; } let books: Book[] = []; export const initializeBooks = (): void => { const filePath = path.join(__dirname, '../data/books.json'); const data = fs.readFileSync(filePath, 'utf-8'); books = JSON.parse(data); }; export const getBooks = (): Book[] => { return books; }; export const findBookById = (id: number): Book | undefined => { return books.find((b) => b.id === id); }; export const addBook = (newBook: Book): void => { books.push(newBook); }; export const deleteBook = (id: number): void => { books = books.filter((b) => b.id !== id); }; export const saveBooks = (): void => { const filePath = path.join(__dirname, '../data/books.json'); fs.writeFileSync(filePath, JSON.stringify(books, null, 2)); }; The initial version of the application is almost ready. Update server.ts code to initiate the database and then add a server startup script in the package.json file. TypeScript /*Path of the file: project-root/src/server.ts*/ import express from 'express'; import dotenv from 'dotenv'; import router from './routes'; import { initializeBooks } from './services/bookService'; dotenv.config(); initializeBooks(); const app = express(); const PORT = process.env.PORT || 3000; app.use(express.json()); app.use('/api', router); app.listen(PORT, () => { console.log(`Server is running on http://localhost:${PORT}`); }); JSON /*Path of the file: project-root/package.json*/ ...rest of file "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "start": "ts-node-dev src/server.ts" }, ...rest of file Finally, start the application by using command npm start. You should see output like below on the screen, and the server should start. App Running Testing the APIs Without Authentication Now that the server is up, you should be able to test the API. Use a tool such as Postman and access the URL http://localhost:3000/api/books to get responses from APIs. You should see a response like the one below: API Call Similarly, you can use API endpoints to update or delete books as well. I have created a postman collection which you should be able to import use inside the postman. You can get at this link. The API for creating new books is http://localhost:3000/api/books, and the API to delete the books is http://localhost:3000/api/books/:id. Implementing JWT Authentication At this point, you are ready to secure the APIs. You will need a list of users who can access the book management APIs. Create a dummy users.json file under the data directory to hold our in-memory users. JSON /*Path of the file: project-root/src/data/users.json*/ [ { "id": 1, "username": "john_doe", "email": "john@example.com", "password": "password1" }, { "id": 2, "username": "jane_doe", "email": "jane@example.com", "password": "password2" } ] Now it is time to create two file userService.ts and userController.ts which will hold login to provide a route to authenticate a user based on username and password. TypeScript /*Path of the file: project-root/src/services/userService.ts*/ import fs from 'fs'; import path from 'path'; interface User { id: number; username: string; email: string; password: string; } let users: User[] = []; export const initializeUsers = (): void => { const filePath = path.join(__dirname, '../data/users.json'); const data = fs.readFileSync(filePath, 'utf-8'); users = JSON.parse(data); }; export const findUserByUsername = (username: string): User | undefined => { return users.find((user) => user.username === username); }; export const generateToken = (user: User): string => { const payload = { id: user.id, username: user.username }; return jwt.sign(payload, process.env.JWT_SECRET || 'secret', { expiresIn: '1h' }); }; TypeScript /*Path of the file: project-root/src/controllers/userController.ts*/ import { Request, Response } from 'express'; import { findUserByUsername, generateToken } from '../services/userService'; export const loginUser = (req: Request, res: Response): void => { const { username, password } = req.body; if (!username || !password) { res.status(400).json({ message: 'Username and password are required' }); return; } const user = findUserByUsername(username); if (!user) { res.status(401).json({ message: 'Invalid username or password' }); return; } if (user.password !== password) { res.status(401).json({ message: 'Invalid username or password' }); return; } const token = generateToken(user); res.json({ token }); }; In the next step, you need to create an authentication middleware function. This function intercepts all the API calls made and validates whether they come from authenticated users or not. Create a directory middleware under src and add file authMiddleware.ts with the code below. TypeScript /*Path of the file: project-root/src/middleware/authMiddleware.ts*/ import { Request, Response, NextFunction } from 'express'; import jwt from 'jsonwebtoken'; export const authMiddleware = (req: Request, res: Response, next: NextFunction): void => { const token = req.header('Authorization')?.split(' ')[1]; if (!token) { res.status(401).json({ message: 'Access Denied. No token provided.' }); return; } try { jwt.verify(token, process.env.JWT_SECRET || 'secret'); next(); } catch (error) { res.status(400).json({ message: 'Invalid Token' }); } }; Now, it's time to incorporate the authentication logic in each API call. Update the routes file to include the authMiddlware in each API call related to book management, as well as add a route related to login. TypeScript /*Path of the file: project-root/src/routes/index.ts*/ import { Router } from 'express'; import { getAllBooks, getBookById, addNewBook, removeBook } from '../controllers/bookController'; import { loginUser } from '../controllers/userController'; import { authMiddleware } from '../middleware/authMiddleware'; const router = Router(); router.post('/login', loginUser); router.get('/books', authMiddleware, getAllBooks); router.get('/books/:id', authMiddleware, getBookById); router.post('/books', authMiddleware, addNewBook); router.delete('/books/:id', authMiddleware, removeBook); export default router; In the final step, initialize the memory user database. Update server.ts file to make them look like the one below. TypeScript /*Path of the file: project-root/src/server.ts*/ import express from 'express'; import dotenv from 'dotenv'; import router from './routes'; import { initializeBooks } from './services/bookService'; import { initializeUsers } from './services/userService'; dotenv.config(); const app = express(); const PORT = process.env.PORT || 3000; initializeBooks(); initializeUsers(); app.use(express.json()); // Middleware to parse JSON app.use('/api', router); app.listen(PORT, () => { console.log(`Server is running on http://localhost:${PORT}`); }); Testing the APIs With Authentication Calling APIs without providing the correct JWT token will now result in the below error from the server. JSON { "message": "Access Denied. No token provided." } Before calling the APIs, you need to authenticate using the URL http://localhost:3000/api/login. Use this URL and provide your username and password. This will give you a valid JWT token, as illustrated below. JWT AUTH You should pass the received JWT to each API and preappend with the word bearer, as highlighted below. This will give you the correct response. Response with JWT Token Conclusion Securing your APIs is the most critical step in modern backend system design. So, congratulations on securing APIs with JWT. JWT makes authenticating APIs stateless and scalable. By leveraging JWT, your Node.js and Express APIs are now better equipped to handle real-world security challenges.

Imagine you're working on a complex puzzle. There are two ways to solve it: The first way: You keep rearranging all the pieces directly on the table, moving them around, and sometimes the pieces you've already arranged get disturbed. This is like traditional imperative programming, where we directly modify data and state as we go. The second way: For each step, you take a picture of your progress, and when you want to try something new, you start with a fresh copy of the last successful attempt. No previous work gets disturbed, and you can always go back to any of the prior states. This is functional programming — where we transform data by creating new copies instead of modifying existing data. Functional programming isn't just another programming style — it's a way of thinking that makes your code more predictable, testable, and often, more readable. In this article, we'll break down functional programming concepts in a way that will make you say, "Ah, now I get it!" What Makes Code "Functional"? Let's break down the core concepts that separate functional programming from traditional imperative (or "primitive") programming. 1. Pure Functions: The Heart of FP In functional programming, pure functions are like vending machines. Given the same input (money and selection), they always return the same output (specific snack). They don't: Keep track of previous purchasesModify anything outside themselvesDepend on external factors Code examples: Plain Text // Impure function - Traditional approach class Calculator { // This variable can be changed by any method, making it unpredictable private int runningTotal = 0; // Impure method - it changes the state of runningTotal public int addToTotal(int number) { runningTotal += number; // Modifying external state return runningTotal; } } // Pure function - Functional approach class BetterCalculator { // Pure method - only works with input parameters // Same inputs will ALWAYS give same outputs public int add(int first, int second) { return first + second; } } // Usage example: Calculator calc = new Calculator(); System.out.println(calc.addToTotal(5)); // Output: 5 System.out.println(calc.addToTotal(5)); // Output: 10 (state changed!) BetterCalculator betterCalc = new BetterCalculator(); System.out.println(betterCalc.add(5, 5)); // Always outputs: 10 System.out.println(betterCalc.add(5, 5)); // Always outputs: 10 2. Immutability: Treat Data Like a Contract In traditional programming, we often modify data directly. In functional programming, we treat data as immutable - once created, it cannot be changed. Instead of modifying existing data, we create new data with the desired changes. Plain Text // Traditional approach - Mutable List public class MutableExample { public static void main(String[] args) { // Creating a mutable list List<String> fruits = new ArrayList<>(); fruits.add("Apple"); fruits.add("Banana"); // Modifying the original list - This can lead to unexpected behaviors fruits.add("Orange"); System.out.println(fruits); // [Apple, Banana, Orange] } } // Functional approach - Immutable List public class ImmutableExample { public static void main(String[] args) { // Creating an immutable list List<String> fruits = List.of("Apple", "Banana"); // Instead of modifying, we create a new list List<String> newFruits = new ArrayList<>(fruits); newFruits.add("Orange"); // Original list remains unchanged System.out.println("Original: " + fruits); // [Apple, Banana] System.out.println("New List: " + newFruits); // [Apple, Banana, Orange] } } 3. Declarative vs. Imperative: The "What" vs. the "How" Traditional programming often focuses on how to do something (step-by-step instructions). Functional programming focuses on what we want to achieve. Plain Text public class NumberProcessing { public static void main(String[] args) { List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5, 6); // Traditional approach (imperative) - Focusing on HOW List<Integer> evenNumbersImperative = new ArrayList<>(); // Step by step instructions for (Integer number : numbers) { if (number % 2 == 0) { evenNumbersImperative.add(number); } } // Functional approach (declarative) - Focusing on WHAT List<Integer> evenNumbersFunctional = numbers.stream() // Just specify what we want: numbers that are even .filter(number -> number % 2 == 0) .collect(Collectors.toList()); System.out.println("Imperative Result: " + evenNumbersImperative); System.out.println("Functional Result: " + evenNumbersFunctional); } } Why Choose Functional Programming? Predictability: Pure functions always produce the same output for the same input, making code behavior more predictable.Testability: Pure functions are easier to test because they don't depend on external state.Debugging: When functions don't modify the external state, bugs are easier to track down.Concurrency: Immutable data and pure functions make concurrent programming safer and more manageable. Common Functional Programming Patterns Here's a quick look at some common patterns you'll see in functional programming: Plain Text public class FunctionalPatterns { public static void main(String[] args) { List<Integer> numbers = Arrays.asList(1, 2, 3, 4, 5); // 1. Map: Transform each number to its doubled value List<Integer> doubled = numbers.stream() .map(number -> number * 2) // transforms each number .collect(Collectors.toList()); System.out.println("Doubled: " + doubled); // [2, 4, 6, 8, 10] // 2. Filter: Keep only even numbers List<Integer> evens = numbers.stream() .filter(number -> number % 2 == 0) // keeps only even numbers .collect(Collectors.toList()); System.out.println("Evens: " + evens); // [2, 4] // 3. Reduce: Sum all numbers int sum = numbers.stream() .reduce(0, (a, b) -> a + b); // combines all numbers into one System.out.println("Sum: " + sum); // 15 } } Conclusion Remember: Just like taking pictures of your puzzle progress, functional programming is about creating clear, traceable transformations of your data. Each step is predictable, reversible, and clean. Start small — try using map, filter, and reduce instead of for loops. Experiment with keeping your data immutable. Soon, you'll find yourself naturally thinking in terms of data transformations rather than step-by-step instructions.

Kubernetes is not new and has been a de-facto standard of deployments and CI/CD at most companies for a while. The goal of this article is to make you familiar with all the terms and jargon that Kubernetes experts use, in approximately 5 minutes! Introduction to Kubernetes Kubernetes provides a scalable framework to manage containers, offering features that span basic cluster architecture to advanced workload orchestration. This piece goes over both the basic and advanced features of Kubernetes. It talks about architecture, resource management, layered security, and networking solutions. It ends with looking at service meshes and persistent storage. Building Blocks Kubernetes operates on a cluster architecture comprising control plane nodes and worker nodes. I sometimes like to refer to "worker nodes" as data planes. The control plane coordinates the cluster, with core components such as: API Server: Manages all cluster operations via RESTful APIsScheduler: Assigns pods to nodes based on resource availability and policiesControllers: Align cluster state with desired configurations (e.g., ReplicaSets, Deployments)etcd: Provides a robust key-value store for all cluster dataThe Data Plane hosts application workloads and includes: Kubelet: Manages pod execution and node operationsKube-proxy: Configures networking rules to connect podsContainer Runtime: Runs containers using tools like containerd, which is open source Refer to the Kubernetes API reference guide for detailed component insights. Resource Management Kubernetes organizes workloads into logical resources such as: Pods: The smallest deployable unit, often hosting one or more containersDeployments: Manage stateless workloads with rolling updates and scalingStatefulSets: Provide persistent storage and ordered scheduling for stateful applicationsDaemonSets: Ensures that system-level pods are set up on all nodes Security Measures Security is a top priority in cloud-native environments, and Kubernetes provides a comprehensive suite of features to address this need. Kubernetes offers several security features, including: RBAC: Role-Based Access Control (RBAC) lets you be very specific about who in the cluster can see what; it is often also referred to as ACLs (Access Control Lists).Network Policies: Network Policies enable you to regulate communication between Pods, adding an extra layer of protection by isolating sensitive workloads.Secrets: Secrets are a safe way to store sensitive credentials, keeping important information private and avoiding accidental exposure. Networking Kubernetes adopts a flat networking model where all pods communicate seamlessly across nodes. Key networking features include: Services: Expose pods to network traffic, enabling internal and external access.Ingress: Manages external HTTP traffic with routing rules.Network Policies: Control ingress and egress traffic for pods, enhancing security. Service Mesh for Microservices Complex microservices often require advanced communication capabilities beyond Kubernetes services. Service meshes like Istio, Linkerd, and Consul provide: Automated mTLS encryption for secure service-to-service commsTraffic routing, observability, and even load balancingSupport for A/B testing, circuit breaking, and traffic splitting These tools eliminate the need for custom-coded communication logic, streamlining development. Persistent Storage for Stateful Workloads Kubernetes supports persistent storage via the Container Storage Interface (CSI), enabling integration with diverse storage backends. This only makes sense though if your application is stateful (or, using StatefulSets). Key resources include: PersistentVolumes (PV): Represent physical or cloud-based storagePersistentVolumeClaims (PVC): Allow workloads to request storage dynamicallyStorageClasses: Simplify storage configuration for diverse workload needs StatefulSets combined with PVCs ensure data durability even during pod rescheduling. Performance: Azure Kubernetes Service As Example Performance really depends on how large the container image that you are running is. Managed solutions like Azure Kubernetes Service provide all of these mentioned above, plus, offer the reliability of Azure behind it. Azure enhanced its infrastructure to reduce the container startup time by pre-caching common base images, so customers can see 15-20x performance gain on cold container startups [1]. Conclusion As I said earlier, unless you are living under a rock, Kubernetes has become an essential tool for organizations embracing cloud-native practices. Its robust, decoupled architecture, combined with strong security features, allows for the efficient deployment and management of containerized applications. For further learning, consult the official Kubernetes documentation. References US11966769B2 - Container instantiation with union file system layer mounts- Google Patents. Hotinger, E. R., Du, B., Antony, S., Lasker, S. M., Garudayagari, S., You, D., Wang, Y., Shah, S., Goff, B. T., Zhang, S., & Llc, M. T. L. (2019, May 23).

TL; DR: The Lean Tech Manifesto With Fabrice Bernhard: Hands-on Agile #65 Join Fabrice Bernhard on how the “Lean Tech Manifesto” solves the challenge of scaling Agile for large organizations and enhances innovation and team autonomy (note: the recording is in English). Lean Tech Manifesto Abstract The release of the Agile Manifesto on February 13th, 2001, marked a revolutionary shift in how tech organizations think about work. By empowering development teams, Agile cut through the red tape in software development and quickly improved innovation speed and software quality. Agile's new and refreshing approach led to its adoption beyond just the scope of a development team, spreading across entire companies far beyond the initial context the manifesto’s original thinkers designed for it. And here lies the problem: the Agile Manifesto was intended for development teams, not for organizations with hundreds or thousands of people. As enthusiasts of Agile, Fabrice and his partner went through phases of excitement and then frustration as they experienced these limitations firsthand while their company grew and our clients became larger. What gave them hope was seeing organizations on both sides of the Pacific, in Japan and California, achieve growth and success almost unmatched while retaining the principles that made the Agile movement so compelling. The “Lean Tech Manifesto” resulted from spending the past 15 years studying these giants and experimenting as they scaled their business. It tries to build on the genius of the original 2001 document but adapt it to a much larger scale. Fabrice shares the connection we identified between Agile and Lean principles and the tech innovations we found the best tech organizations adopt to distribute work and maintain team autonomy. Meet Fabrice Bernhard Fabrice Bernhard is the co-author of The Lean Tech Manifesto and the Group CTO of Theodo, a leading technology consultancy he cofounded with Benoît Charles-Lavauzelle and scaled from 10 people in 2012 to 700 people in 2022. Based in Paris, London and Casablanca, Theodo uses Agile, DevOps, and Lean to build transformational tech products for clients all over the world, including global companies — such as VF Corporation, Raytheon Technologies, SMBC, Biogen, Colas, Tarkett, Dior, Safran, BNP Paribas, Allianz, and SG — and leading tech scale-ups — such as ContentSquare, ManoMano, and Qonto. Fabrice is an expert in technology and large-scale transformations and has contributed to multiple startups scaling more sustainably with Lean thinking. He has been invited to share his experience at international conferences, including the Lean Summit, DevopsDays, and CraftConf. The Theodo story has been featured in multiple articles and in the book Learning to Scale at Theodo Group. Fabrice is also the co-founder of the Paris DevOps meetup and an active YPO member. He studied at École Polytechnique and ETH Zürich and lives with his two sons in London. Connect with Fabrice Bernhard on LinkedIn. Video Watch the recording of Fabrice Bernhard’s The Lean Tech Manifesto session now:

By Stefan Wolpers CORE