DZone Spotlight

Developers may be aware of the lifecycle of service instances when using dependency injection, but many don’t fully grasp how it works. You can find numerous articles online that clarify these concepts, but they often just reiterate definitions that you might already know. Let me illustrate with a detailed example that simplifies the explanation. When implementing dependency injection, developers have three options that determine the lifecycle of the instances: Singleton Scoped Transient While most developers recognize these terms, a significant number struggle to determine which option to choose for a service's lifetime. Definitions Let me start with definitions: Singleton lifetime service instances are created once per application from the service container. A single instance will serve all subsequent requests. Singleton services are disposed of at the end of the application (i.e., upon application restart). Transient lifetime service instances are created per request from the service container. Transient services are disposed of at the end of the request. Scoped lifetime service instances are created once per client request. Transient services are disposed of at the end of the request. When to Use Singleton - When you want to use single instances of services throughout the life cycle of the application Transient - When you want to use individual instances of services within the client request Scoped - When you want to use a single instance of service for each request What is a client request? In very simple words, you can consider it as an API/REST call coming to your application by button clicks of the user to get the response. Don’t worry, let's understand with an example. Example First, let's create interfaces/services and classes: C# // we are declaring 3 services as below Public interface ISingleton Public interface ITransient Public interface IScoped Now let's write the implementation for each service Interface/service created above. We will try to understand the concept by trying to update the callMeSingleton, callMeTransient, and callMeScoped variable. Singleton class implementation: C# class SingletonImplementation: ISingleton { var callMeSingleton = "" // other implementation public SetSingleton(string value) { callMeSingleton = value; } // other implementation } Transient class implementation: C# class TransientImplementation: ITransient { var callMeTransient = "" // other implementation public SetTransient(string value) { callMeTransient = value; } // other implementation } Scoped class implementation: C# class ScopedImplementation: IScoped { var callMeScoped = "" //other implementation public SetScoped(string value) { callMeScoped = value; } //other implementation } Let's register (ConfigureServices) with DI (Dependency Injection) to decide the life cycle of each service instance: C# services.AddSingleton<ISingleton, SingletonImplementation>(); services.AddTransient<ITransient , TransientImplementation>(); services.AddScoped<IScoped , ScopedImplementation>(); Let's use/call these services from 3 different classes (ClassA, ClassB, and ClassC) to understand the life cycle of each service: ClassA: C# public class ClassA { private ISingleton _singleton; //constructor to instantiate 3 different services we creates public ClassA(ISingleton singleton, ITransient _transient, IScoped _scoped) { _singleton = singleton; } public void UpdateSingletonFromClassA() { _singleton.SetSingleton("I am from ClassA"); } public void UpdateTransientFromClassA() { _transient.SetTransient("I am from ClassA"); } public void UpdateScopedFromClassA() { _scoped.SetScoped("I am from ClassA"); } // other implementation } ClassB: C# public class ClassB { private ISingleton _singleton; //constructor to instantiate 3 different services we creates public ClassB(ISingleton singleton, ITransient _transient, IScoped _scoped) { _singleton = singleton; } public void UpdateSingletonFromClassB() { _singleton.SetSingleton("I am from ClassB"); } public void UpdateTransientFromClassB() { _transient.SetTransient("I am from ClassB"); } public void UpdateScopedFromClassB() { _scoped.SetScoped("I am from ClassB"); } // other implementation } ClassC: C# public class ClassC { private ISingleton _singleton; //constructor to instantiate 3 different services we creates public ClassC(ISingleton singleton, ITransient _transient, IScoped _scoped) { _singleton = singleton; } public void UpdateSingletonFromClassC() { _singleton.SetSingleton("I am from ClassC"); } public void UpdateTransientFromClassC() { _transient.SetTransient("I am from ClassC"); } public void UpdateScopedFromClassC() { _scoped.SetScoped("I am from ClassC"); } // other implementation } Analysis Let's analyze the results and behavior for each life cycle one by one from the above implementation: Singleton All the classes (ClassA, ClassB, and ClassC) will use the same single instance of the SingletonImplementation class throughout the lifecycle of the application. This means that properties, fields, and operations of the SingletonImplementation class will be shared among instances used on all calling classes. Any updates to properties or fields will override previous changes. For example, in the code above, ClassA, ClassB, and ClassC are all utilizing the SingletonImplementation service as a singleton instance and calling SetSingleton to update the callMeSingleton variable. In this case, there will be a single value of the callMeSingleton variable for all requests trying to access this property. Whichever class accesses it last to update will override the value of callMeSingleton. ClassA - It will have its same instance as other classes for service TransientImplementation. ClassB - It will have its same instance as other classes for service TransientImplementation. ClassC - It will have its same instance as other classes for service TransientImplementation. ClassA, ClassB, and ClassC are updating the same instance of the SingletonImplementation class, which will override the value of callMeSingleton. Therefore, be careful when setting or updating properties in the singleton service implementation. Singleton services are disposed of at the end of the application (i.e., upon application restart). Transient All the classes (ClassA, ClassB, and ClassC) will use their individual instances of the TransientImplementation class. This means that if one class calls for properties, fields, or operations of the TransientImplementation class, it will only update or override its individual instance values. Any updates to properties or fields are not shared among other instances of TransientImplementation. Let's understand: ClassA - It will have its own instance of service of TransientImplementation. ClassB - It will have its own instance of service of TransientImplementation. ClassC - It will have its own instance of service of TransientImplementation. Let's say you have a ClassD which is calling transient service from ClassA, ClassB, and ClassC instances. In this case, each class instance would be treated as different/separate instance and each class would have its own value of callMeTransient. Read the inline comments below for ClassD: C# public ClassD { // other implementation // Below line of code will update the value of callMeTransient to "I am from ClassA" for the intance of ClassA only. // And it will not be changed by any next calls from Class B or B class ClassA.UpdateTransientFromClassA(); // Below line of code will update the value of callMeTransient to "I am from ClassB" for the intance of ClassB only. // And it will neither override the value for calssA instance nor will be changed by next call from Class C ClassB.UpdateTransientFromClassB(); // Below line of code will update the value of callMeTransient to "I am from ClassC" for the intance of ClassC only. // And it will neither override the value for calssA and classB instance nor will be changed by any next call from any other class. ClassC.UpdateTransientFromClassC(); // other implementation } Transient services are disposed at the end of each request. Use Transient when you want a state less behavior within the request. Scoped All the classes (ClassA, ClassB, and ClassC) will be using single instances of ScopedImplementation class for each request. This means that calls for properties/fields/operations on ScopedImplementation class will happen on single instance with in the scope of request. Any updates of properties/fields will be shared among other classes. Let's understand: ClassA - It will have its instance of service of TransientImplementation. ClassB - It will have its same instance of service of TransientImplementation as ClassA. ClassC - It will have its same instance of service of TransientImplementation as ClassA and ClassB. Let's say you have a ClassD which is calling scoped service from ClassA, ClassB, and ClassC instances. In this case, each class will have single instance of ScopedImplementation class. Read the inline comments for ClassD below. C# public class ClassD { // other implementation // Below code will update the value of callMeScoped to "I am from ClassA" for the instance of ClassA // But as it is Scoped life cycle so it is holding single instance ScopedImplementation of // Then it can be overridden by next call from ClassB or ClassC ClassA.UpdateScopedFromClassA(); // Below code will update the value of callMeScoped to "I am from ClassB" for single instance ScopedImplementation // And it will override the value of callMeScoped for classA instance too. ClassB.UpdateScopedFromClassB(); // Now if Class A will perform any operation on ScopedImplementation, // it will use the latest properties/field values which are overridden by classB. // Below code will update the value of callMeScoped to "I am from ClassC" // And it will override the value of callMeScoped for classA and ClassB instance too. ClassC.UpdateScopedFromClassC(); // now if Class B or Class A will perform any operation on ScopedImplementation , it will use the latest properties/field values which are overridden by classC // other implementation } Scoped services are disposed at the end of each request. Use Scoped when you want a stateless behavior between individual requests. Trivia Time The lifecycle of a service can be overridden by a parent service where it gets initialized. Confused? Let me explain: Let's take the same example from above classes and initialize the Transient and Scoped services from SingletonImplementation (which is a singleton) as below. That would initiate the ITransient and IScoped services and overwrite the lifecycle of these to singleton life cycle as parent service. In this case your application would not have any Transient or Scoped services (considering you just have these 3 services we were using in our examples). Read through the lines in the below code: C# public class SingletonImplementation: ISingleton { // constructor to add initialize the services. private readonly ITransient _transient private readonly IScoped _scoped SingletonImplementation(ITransient transient, IScoped scoped) { _transient = transient; // Now _transient would behave as singleton service irrespective of how it was registered as Transient _scoped = scoped; // now scoped would behave as singleton service irrespective of it being registered as Scoped } var callMeSingleton = "" // other implementation } Summary I hope the above article is helpful in understanding the topic. I would recommend try it yourself with the context set above and you will never be confused again. Singleton is the easiest to understand because once you create its instance, it will be shared across applications throughout the lifecycle of the application. On the similar lines of Singleton, Scoped instances mimic the same behavior but only throughout the lifecycle of a request across application. Transient is totally stateless, for each request and each class instance will hold its own instance of serivice.

"What's the point of agents? Why use something like AutoGen when I can code it myself?" Sounds familiar? If you have ever thought about this, you're not alone. I hear this all the time. And I know exactly where this is coming from. In this post, we’re going to dive into the world of agentic design patterns — a powerful approach that goes beyond simple code. These patterns can help you build AI systems where agents don’t just complete tasks; they delegate, verify, and even work together to tackle complex challenges. Ready to level up your AI game? Let’s go! The Two-Agent Pattern Scenario: Replacing a UI Form With a Chatbot Let’s start with a common problem: you want to replace a form in your app with a chatbot. Simple, right? The user should be able to create or modify data using natural language. The user should be able to say things like, “I want to create a new account,” and the chatbot should ask all the relevant questions, fill in the form, and submit it — all without losing track of what’s already been filled. Challenge This might seem like an easy example, right? I mean, we can simply shove all the fields we need to ask the user into a system prompt. Well, this approach might work but you'll notice things get out of hand really quickly. A single agent might struggle to manage and keep track of long, complex conversations. It might not be able to keep up. As the conversation drags on, the chatbot might forget which questions it’s already asked or fail to gather all the necessary information. This can lead to a frustrating user experience, where fields are missed or the chatbot asks redundant questions. Solution This is where the two-agent pattern comes in. Instead of letting our chatbot agent respond to the user directly, we make it have an internal conversation with a "companion" agent. This helps delegate some responsibility from the "primary" agent over to the "companion." In our example, we can divide the work between the following two agents: Chatbot agent: Responsible for carrying the conversation with the user, dealing with prompt injection, and preventing the conversation from getting derailed Form agent: Responsible for remembering form fields and tracking progress In this solution, whenever our chatbot agent gets a message from the user, it first forwards it to the form agent to identify if the user has provided new information. This information is stored in the form agent's memory. The form agent then calculates the fields that are pending and nudges the chatbot agent to ask those questions. And since the form agent isn't looking at the entire conversation at once, it doesn't really suffer from problems arising from long and complex conversation histories. The Reflection Pattern Scenario: RAG Chatbot Using a Knowledge Base For our second pattern, let's assume that you’ve built a chatbot that answers user questions by pulling information from a knowledge base. The chatbot is used for important tasks, like providing policy or legal advice. It’s critical that the information is accurate. We don't really want our chatbot misquoting facts and hallucinating responses. Challenge Anyone who has tried to build a RAG-powered chatbot knows the challenges that come alongside it. Chatbots can sometimes give incorrect or irrelevant answers, especially when the question is complex. If your chatbot responds with outdated or inaccurate info, it could lead to serious consequences. So is there a way to prevent the chatbot from going rogue? Can we somehow fact-check each response our chatbot gives us? Solution The two-agent pattern to the rescue — but this time we use a specialized version of the two-agent pattern called reflection. Here we introduce a secondary "verifier" agent. Before the chatbot sends its response, the verifier agent can check for a couple of things: Groundedness: Is the answer based on the chunks extracted from the retrieval pipeline, or is the chatbot just hallucinating information? Relevance: Is the answer (and the chunks retrieved) actually relevant to the user's question? If the verifier finds issues, it can instruct the chatbot to retry or adjust the response. It can even go ahead to mark certain chunks as irrelevant to prevent them from being used again. This system keeps the conversation accurate and relevant, especially in high-stakes environments. The Sequential Chat Pattern Scenario: Blog Creation Workflow Let’s say you’re creating a blog post — just like this one. The process involves several stages: researching your topic, identifying key talking points, and creating a storyline to tie it all together. Each stage is essential to creating a polished final product. Challenge Sure, you could try to generate the entire video script using a single, big, fancy AI prompt. But you know what you’ll get? Something generic, flat, and just... meh. Not exactly the engaging content that’s going to blow your audience away. The research might not go deep enough, or the storyline might feel off. That’s because the different stages require different skills. And here’s the kicker: you don’t want one model doing everything! You’d want a precise, fact-checking model for your research phase while using something more creative to draft a storyline. Using the same AI for both jobs just doesn’t cut it. Solution Here's where the sequential chat pattern comes in. Instead of a one-size-fits-all approach, you can break the workflow into distinct steps, assigning each one to a specialized agent. Need research done by consuming half the internet? Assign it to a researcher agent. Want to extract the key points from those research notes? There's an agent for that too! And when it's time to get creative, well... you get the point. It's important to remember that these agents are not technically conversing with each other. We simply take the output of one agent and pass it on to the next. Pretty much like prompt chaining. So why not just use prompt chaining? Why agents? Well, it is because agents are composable. Agents as Composable Components Scenario: Improving Our Blog Creation Workflow Alright, this isn’t exactly a “pattern,” but it’s what makes agents so exciting. Let’s jump back to our blog creation workflow example. Imagine your talking point analyzer agent isn’t hitting the mark. It’s not aligned with what your audience wants to hear, and the talking points are kinda off. Can we make it better? You bet! Solution What if we bring in the reflection pattern here? We could add a reflection agent that compares the talking points with what’s worked in previous blog posts — real audience data. This grounding agent ensures that your talking points are always in tune with what your viewers love. But wait. Does this mean we have to change our entire workflow? No. Not really. Because agents are composable, to the outside world, everything still works exactly the same! No one needs to know that behind the scenes, you’ve supercharged your workflow. It’s like upgrading the engine of a car without anyone noticing, but suddenly it runs like a dream! The Group Chat Pattern Scenario: Building a Coding-Assistance Chatbot Alright, picture this: you’re building a chatbot that can help developers with all kinds of coding tasks—writing tests, explaining code, and even building brand-new features. The user can throw any coding question at your bot, and boom, it handles it! Naturally, you’d think, “Let’s create one agent for each task.” One for writing tests, one for code explanations, and another for feature generation. Easy enough, right? But wait, there's a catch. Challenge Here’s where things get tricky. How do you manage something this complex? You don’t know what kind of question the user will throw your way. Should the chatbot activate the test-writing agent or maybe the code-explainer? What if the user asks for a new feature — do you need both? And here’s the kicker: some requests need multiple agents to work together. For example, creating a new feature isn’t just about generating code. First, the bot needs to understand the existing codebase before writing anything new. So now, the agents have to team up, but who’s going first, and who’s helping who? It’s a lot to handle! Solution The group chat pattern to the rescue. Let's introduce a planner agent into the mix. This agent acts like the ultimate coordinator, deciding which agents should tackle the task and in what order. If a user asks for a new feature, the planner first calls the code-explainer agent to understand the existing code, then hands it off to the feature-generation agent to write the new code. Easy, right? But here’s the fun part — the planner doesn’t just set things up and leave. It can adapt on the go! If there’s an error in the generated code, it loops back to the coding agent for another round. The planner ensures that everything runs smoothly, with agents working together like an all-star team, delivering exactly what the user needs, no matter how complex the task. Video Conclusion To wrap things up, agentic design patterns are more than just fancy names — they’re practical tools that can simplify your AI workflows, reduce errors, and help you build smarter, more flexible systems. Whether you're delegating tasks, verifying accuracy, or coordinating complex actions, these patterns have got you covered. But the one thing you should absolutely take home is that agents are composable. They can evolve over time, handling increasingly complex tasks with ease. So, ready to dive deeper? Here are some helpful links to take your journey forward: AutoGen Tutorial [Video] Conversational patterns using AutoGen [Documentation]

In this blog, you will learn how to get started with jOOQ, Liquibase, and Testcontainers. You will create a basic Spring Boot application and integrate the aforementioned techniques including a test setup. Furthermore, you will use Spring Boot Docker Compose support to run the application in no time. Enjoy! 1. Introduction The starting point for this blog was to get more acquainted with jOOQ, a database-first approach instead of using an Object Relation Mapper (ORM) framework. Being able to just write SQL including typesafety is very appealing and interesting. However, during the setup of the application, some extra requirements popped up. Note that these requirements are my own requirements and choices — these are not imposed by jOOQ. Liquibase needs to be used for creating the database tables. PostgreSQL has to be used as a database. The Maven plugin testcontainers-jooq-codegen-maven-plugin has to be used for generating jOOQ code. Testcontainers should be used for integration tests. Besides that, the application should be accessible via a Rest API defined with an OpenAPI specification and the code should be generated by means of the openapi-generator-maven-plugin. This will not be explained further in this blog, but you can read more about it in a previous blog if you are interested. In the end, you also want a running application with a "real" database. That is where the Spring Boot Docker Compose support will help you. Sources used in this blog are available on GitHub. 2. Prerequisites Quite some technologies are used in this blog. You do not need to be an expert in all of them, but at least you need to know what is used for and have some basic knowledge about it. Prerequisites are: Basic knowledge of Java, Java 21 is used Basic knowledge of Spring Boot Basic knowledge of Liquibase — more information about Liquibase can be found in a previous blog Basic knowledge of Testcontainers — more information about Testcontainers can be found in a previous blog Basic knowledge of OpenAPI if you want to dive into the source code 3. Application Setup In order to get started with a Spring Boot application, you navigate to Spring Initializr. You add the Spring Web dependency and you are ready to go. The application will be able to do three things: Create a customer Retrieve a single customer Retrieve all customers A customer consists of an ID, a first name, a last name, and a country, represented by the Customer class. Java public class Customer { private long id; private String firstName, lastName, country; // Getters and setters left out for brevity } The OpenAPI specification will define the REST endpoints, and based on this specification, the openapi-generator-maven-plugin will generate code for you. An interface CustomersApi is generated for the controller which you need to implement. Java @RestController public class CustomerController implements CustomersApi { @Override public ResponseEntity<Void> createCustomer(Customer apiCustomer) { // to be implemented } @Override public ResponseEntity<List<CustomerFullData>> getCustomers() { // to be implemented return null; } @Override public ResponseEntity<CustomerFullData> getCustomer(Long customerId) { // to be implemented return null; } } 4. Add Liquibase You need database scripts for creating the tables because jOOQ uses a database-first approach. In order to create the database, Liquibase will be used. Add the Liquibase dependency to the pom. XML <dependency> <groupId>org.liquibase</groupId> <artifactId>liquibase-core</artifactId> </dependency> There are several ways to set up and organize Liquibase. In this blog, you will make use of a root changelog file and several separate changelog files for the database. What does this look like? Add the root changelog file to src/main/resources/db/changelog. It only mentions looking in the directory migration for the real changelogs. XML <?xml version="1.0" encoding="UTF-8"?> <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.1.xsd"> <includeAll path="./migration" relativeToChangelogFile="true"/> </databaseChangeLog> Add the changelogs in the directory src/main/resources/db/changelog/migration. Two changelogs are added just to demonstrate how a database migration script can be added. The changelog db.changelog-1.0.xml creates the customer table with the id, first name, and last name columns. XML <?xml version="1.0" encoding="UTF-8"?> <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.1.xsd"> <changeSet author="gunter" id="changelog-1.0"> <createTable tableName="customer"> <column name="id" type="serial" autoIncrement="true"> <constraints nullable="false" primaryKey="true"/> </column> <column name="first_name" type="varchar(255)"> <constraints nullable="false"/> </column> <column name="last_name" type="varchar(255)"> <constraints nullable="false"/> </column> </createTable> <rollback> <dropTable tableName="customer"/> </rollback> </changeSet> </databaseChangeLog> The db.changelog-2.0.xml adds the country column to the customer table. XML <?xml version="1.0" encoding="UTF-8"?> <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.1.xsd"> <changeSet author="gunter" id="changelog-2.0"> <addColumn tableName="customer"> <column name="country" type="varchar(255)"/> </addColumn> <rollback> <dropColumn tableName="customer"> <column name="country" type="varchar(255)"/> </dropColumn> </rollback> </changeSet> </databaseChangeLog> Finally, add the following to the application.properties file. Properties files spring.liquibase.change-log=classpath:db/changelog/db.changelog-root.xml 5. Add jOOQ Code Generation In order to let jOOQ generate code, you need a running instance of the database including your tables. This is where Testcontainers come into play. Testcontainers are typically used for integration tests with a database, but it can also be used as a database for generating the jOOQ code. This can be done with the help of the testcontainers-jooq-codegen-maven-plugin. The documentation of the plugin can be found here. Add the following to your pom. There is quite a lot to see here: Two dependencies for PostgreSQL are needed: one for Testcontainers and one for the driver. In the configuration section, you define which type of database you use and which version should be used for the container image. In the jOOQ generator parameters, you define the tables that should be included or excluded. Note that the Liquibase-specific tables (DATABASECHANGELOG and DATABASECHANGELOGLOCK) are excluded. In the jOOQ generator parameters, you define in which package the generated code should be located. Also note that with tag liquibase, the plugin knows that Liquibase is being used with some default settings. XML <build> <plugins> <plugin> <groupId>org.testcontainers</groupId> <artifactId>testcontainers-jooq-codegen-maven-plugin</artifactId> <version>${testcontainers-jooq-codegen-maven-plugin.version}</version> <dependencies> <dependency> <groupId>org.testcontainers</groupId> <artifactId>postgresql</artifactId> <version>${testcontainers.version}</version> </dependency> <dependency> <groupId>org.postgresql</groupId> <artifactId>postgresql</artifactId> <version>${postgresql.version}</version> </dependency> </dependencies> <executions> <execution> <id>generate-jooq-sources</id> <goals> <goal>generate</goal> </goals> <phase>generate-sources</phase> <configuration> <database> <type>POSTGRES</type> <containerImage>postgres:15-alpine</containerImage> <!-- optional --> </database> <liquibase/> <!-- Generator parameters --> <jooq> <generator> <database> <includes>.*</includes> <excludes>DATABASECHANGELOG.*</excludes> <inputSchema>public</inputSchema> </database> <target> <packageName>com.mydeveloperplanet.myjooqplanet.jooq</packageName> <directory>target/generated-sources/jooq</directory> </target> </generator> </jooq> </configuration> </execution> </executions> </plugin> </plugins> </build> Generate the code: Shell $ mvn generate-sources In the console output, you will notice that: A Testcontainer is started. The Liquibase migration scripts are applied. The jOOQ code is generated. Check the target/generated-sources/jooq directory. The Tables class contains the tables. Java public class Tables { /** * The table <code>public.customer</code>. */ public static final Customer CUSTOMER = Customer.CUSTOMER; } The Keys class contains the primary keys. Java public class Keys { // ------------------------------------------------------------------------- // UNIQUE and PRIMARY KEY definitions // ------------------------------------------------------------------------- public static final UniqueKey<CustomerRecord> CUSTOMER_PKEY = Internal.createUniqueKey(Customer.CUSTOMER, DSL.name("customer_pkey"), new TableField[] { Customer.CUSTOMER.ID }, true); } A Customer class contains the table definition. Java public class Customer extends TableImpl<CustomerRecord> { private static final long serialVersionUID = 1L; /** * The reference instance of <code>public.customer</code> */ public static final Customer CUSTOMER = new Customer(); /** * The class holding records for this type */ @Override public Class<CustomerRecord> getRecordType() { return CustomerRecord.class; } // And a lot more } A CustomerRecord defines a record in the table. Java public class CustomerRecord extends UpdatableRecordImpl<CustomerRecord> implements Record4<Integer, String, String, String> { private static final long serialVersionUID = 1L; /** * Setter for <code>public.customer.id</code>. */ public void setId(Integer value) { set(0, value); } /** * Getter for <code>public.customer.id</code>. */ public Integer getId() { return (Integer) get(0); } // And a lot more } 6. Add Repository Now it is time to connect some dots. First, you need a repository for the database access. The only thing you need to do is to inject a DSLContext which is auto-configured by Spring Boot. Add the following dependency to the pom. XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-jooq</artifactId> </dependency> With the DSLContext, you can implement the required repository methods, making use of the generated jOOQ code. It is pretty straightforward and very readable. Java @Repository public class CustomerRepository { private final DSLContext create; CustomerRepository(DSLContext create) { this.create = create; } public void addCustomer(final Customer customer) { create.insertInto(CUSTOMER, CUSTOMER.FIRST_NAME, CUSTOMER.LAST_NAME, CUSTOMER.COUNTRY) .values(customer.getFirstName(), customer.getLastName(), customer.getCountry()) .execute(); } public CustomerRecord getCustomer(int customerId) { return create.selectFrom(CUSTOMER).where(CUSTOMER.ID.eq(customerId)).fetchOne(Records.mapping(CustomerRecord::new)); } public List<CustomerRecord> getAllCustomers() { return create.selectFrom(CUSTOMER) .fetch(Records.mapping(CustomerRecord::new)); } } Build the application: Shell $ mvn clean verify 7. Complete Controller The last thing to do is to implement the controller. You inject the repository and implement the interface methods. Java @RestController public class CustomerController implements CustomersApi { public final CustomerRepository customerRepository; public CustomerController(CustomerRepository customerRepository) { this.customerRepository = customerRepository; } @Override public ResponseEntity<Void> createCustomer(Customer apiCustomer) { com.mydeveloperplanet.myjooqplanet.Customer customer = new com.mydeveloperplanet.myjooqplanet.Customer(); customer.setFirstName(apiCustomer.getFirstName()); customer.setLastName(apiCustomer.getLastName()); customer.setCountry(apiCustomer.getCountry()); customerRepository.addCustomer(customer); return ResponseEntity.ok().build(); } @Override public ResponseEntity<List<CustomerFullData>> getCustomers() { List<CustomerRecord> customers = customerRepository.getAllCustomers(); List<CustomerFullData> convertedCustomers = convertToCustomerFullData(customers); return ResponseEntity.ok(convertedCustomers); } @Override public ResponseEntity<CustomerFullData> getCustomer(Long customerId) { CustomerRecord customer = customerRepository.getCustomer(customerId.intValue()); return ResponseEntity.ok(repoToApi(customer)); } ... } Build the application: Shell $ mvn clean verify 8. Use Testcontainers for Integration Test Testcontainers are used for generating the jOOQ code, but you can also use it for your integration test. The test uses: @Testcontainers to indicate that this test requires Testcontainers @SpringBootTest to start the Spring Boot application WebTestClient in order to send a request to the application @Container in order to ensure that Testcontainers will manage the lifecycle of the Testcontainer @ServiceConnection in order for Spring Boot to use the default configuration to connect to the Testcontainer — Note that nowhere any database configuration has been added in application.properties. Java @Testcontainers @SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) class MyJooqPlanetApplicationTests { @Autowired private WebTestClient webTestClient; @Container @ServiceConnection static PostgreSQLContainer<?> postgreSQLContainer = new PostgreSQLContainer<>(DockerImageName.parse("postgres:15-alpine")); @Test void whenCreateCustomer_thenReturnSuccess() throws Exception { String body = """ { "firstName": "John", "lastName": "Doe", "country": "Belgium" } """; webTestClient .post() .uri("/customers") .contentType(MediaType.APPLICATION_JSON) .bodyValue(body) .exchange() .expectStatus() .is2xxSuccessful(); } @Test void givenCustomer_whenRetrieveAllCustomers_thenReturnSuccess() throws Exception { String body = """ { "firstName": "John", "lastName": "Doe", "country": "Belgium" } """; webTestClient .post() .uri("/customers") .contentType(MediaType.APPLICATION_JSON) .bodyValue(body) .exchange() .expectStatus() .is2xxSuccessful(); webTestClient .get() .uri("/customers") .exchange() .expectStatus() .is2xxSuccessful() .expectHeader() .contentType(MediaType.APPLICATION_JSON) .expectBody() .jsonPath("$[0].customerId").isEqualTo(1) .jsonPath("$[0].firstName").isEqualTo("John") .jsonPath("$[0].lastName").isEqualTo("Doe") .jsonPath("$[0].country").isEqualTo("Belgium"); } } In order to be able to run this test, you need to add the following test dependencies to the pom. XML <!-- Needed for the testcontainers integration test --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-testcontainers</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>junit-jupiter</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.testcontainers</groupId> <artifactId>postgresql</artifactId> <scope>test</scope> </dependency> <!-- Needed for WebTestClient --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webflux</artifactId> <scope>test</scope> </dependency> In order to run the application, you need to add the following dependencies to the pom. XML <!-- Needed for database access --> <dependency> <groupId>org.postgresql</groupId> <artifactId>postgresql</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>org.liquibase</groupId> <artifactId>liquibase-core</artifactId> <scope>runtime</scope> </dependency> Run the test. Shell $ mvn clean verify 9. Run Application You have built an application, code is generated, and an integration test is successful. But now you also want to run the application. And you will need to have a running database for that. This can easily be accomplished if you add the spring-boot-docker-compose dependency to your pom. XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-docker-compose</artifactId> <scope>runtime</scope> <optional>true</optional> </dependency> Add a compose.yaml to the root of your repository which contains a definition for your database setup. YAML services: postgres: image: 'postgres:15-alpine' environment: - 'POSTGRES_DB=mydatabase' - 'POSTGRES_PASSWORD=secret' - 'POSTGRES_USER=myuser' labels: - "org.springframework.boot.service-connection=postgres" ports: - '5432' Start the application. Shell $ mvn spring-boot:run Notice that the PostgreSQL container is started in the console logs. Shell 2024-07-06T15:59:52.037+02:00 INFO 78905 --- [MyJooqPlanet] [ main] .s.b.d.c.l.DockerComposeLifecycleManager : Using Docker Compose file '/<home directory>/myjooqplanet/compose.yaml' 2024-07-06T15:59:52.276+02:00 INFO 78905 --- [MyJooqPlanet] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container myjooqplanet-postgres-1 Created 2024-07-06T15:59:52.277+02:00 INFO 78905 --- [MyJooqPlanet] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container myjooqplanet-postgres-1 Starting 2024-07-06T15:59:52.610+02:00 INFO 78905 --- [MyJooqPlanet] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container myjooqplanet-postgres-1 Started 2024-07-06T15:59:52.610+02:00 INFO 78905 --- [MyJooqPlanet] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container myjooqplanet-postgres-1 Waiting 2024-07-06T15:59:53.114+02:00 INFO 78905 --- [MyJooqPlanet] [utReader-stderr] o.s.boot.docker.compose.core.DockerCli : Container myjooqplanet-postgres-1 Healthy Create a customer. Shell $ curl -X POST http://localhost:8080/customers \ -H 'Content-Type: application/json' \ -d '{"firstName": "John", "lastName": "Doe", "country": "Belgium" }' Retrieve the customer. Shell $ curl http://localhost:8080/customers/1 {"firstName":"John","lastName":"Doe","country":"Belgium","customerId":1} Retrieve all customers. Shell $ curl http://localhost:8080/customers [{"firstName":"John","lastName":"Doe","country":"Belgium","customerId":1}] 10. Conclusion In this blog, you learned how to integrate a Spring Boot Application with jOOQ, Liquibase, and Testcontainers. You used Testcontainers for an integration test, and you used Spring Boot Docker Compose in order to run your application with a database out-of-the-box. Cool stuff!

By Gunter Rotsaert CORE

It may happen that we create an index, but PostgreSQL doesn’t use it. What could be the reason, and how can we fix it? We identified 11 distinct scenarios. Read on to find out. Important Things Indexes may be tricky. We already covered how they work in another article. Let’s quickly recap the important parts of how they work. How Indexes Work The B-tree is a self-balancing tree data structure that maintains a sorted order of entries, allowing for efficient searches, insertions, and deletions in logarithmic time — unlike regular heap tables, which operate in linear time. B-trees are an extension of binary search trees, with the key distinction that they can have more than two children. B-trees in PostgreSQL satisfy the following: Every node has at most 3 children. Every node (except for the root and leaves) has at least 2 children. The root node has at least two children (unless the root is a leaf). Leaves are on the same level. Nodes on the same level have links to siblings. Each key in the leaf points to a particular TID. Page number 0 of the index holds metadata and points to the tree root. We can see a sample B-tree below: When inserting a key into a full node, the node must be split into two smaller nodes, with the key being moved up to the parent. This process may trigger further splits at the parent level, causing the change to propagate up the tree. Likewise, during key deletion, the tree remains balanced by merging nodes or redistributing keys among sibling nodes. For searching, we can apply the binary search algorithm. Suppose we're searching for a value that appears in the tree only once. Here's how we would navigate the tree while looking for 62: You can see that we just traverse the nodes top-down, check the keys, and finally land at a value. A similar algorithm would work for a value that doesn’t exist in the tree. This is more obvious when we look for all the values that are greater than 0: How Database Estimates the Cost The index cost consists of the start-up cost and the run cost. The start-up cost depends on the number of tuples in the index and the height of the index. It is specified as cpu_operator_cost multiplied by the ceiling of the base-2 logarithm of the number of tuples plus the height of the tree multiplied by 50. The run cost is the sum of the CPU cost (for both the table and the index) and the I/O cost (again, for the table and the index). The CPU cost for the index is calculated as the sum of constants cpu_index_tuple_cost and qual_op_cost (which are 0.005 and 0.0025 respectively) multiplied by the number of tuples in the index multiplied by the selectivity of the index. Selectivity of the index is the estimate of how many values inside the index would match the filtering criteria. It is calculated based on the histogram that captures the counts of the values. The idea is to divide the column’s values into groups of approximately equal populations to build the histogram. Once we have such a histogram, we can easily specify how many values should match the filter. The CPU cost for the table is calculated as the cpu_tuple_cost (constant 0.01) multiplied by the number of tuples in the table multiplied by the selectivity. Notice that this assumes that we’ll need to access each tuple from the table even if the index is covering. The I/O cost for the index is calculated as the number of pages in the index multiplied by the selectivity multiplied by the random_page_cost which is set to 4.0 by default. Finally, the I/O cost for the table is calculated as the sum of the number of pages multiplied by the random_page_cost and the index correlation multiplied by the difference between the worst-case I/O cost and the best-case I/O cost. The correlation indicates how much the order of tuples in the index is aligned with the order of tuples in the table. It can be between -1.0 and 1.0. All these calculations show that index operations can be beneficial if we extract only a subset of rows. It’s no surprise that the index scans will be even more expensive than the sequential scans, but the index seeks will be much faster. Index Construction Indexes can be built on multiple columns. The order of columns matters, the same as the order of values in the column (whether it’s ascending or descending). The index requires a way to sort values. The B-tree index has an inherent capability of value organization within certain data types, a feature we can exploit quite readily for primary keys or GUIDs as the order is typically immutable and predictable in terms of such scenarios due to their "built-in" nature which aligns with increasing values. However, when dealing with complex custom datatypes — say an object containing multiple attributes of varying data types — it’s not always apparent how these might be ordered within a B-tree index given that they do not have an inherent ordering characteristic, unlike primitive (basic) or standardized composite data types such as integers and structured arrays. Consequently, for situations where we deal with complex datatypes — consider the two-dimensional points in space that lack any natural order due to their multi-attribute nature — one might find it necessary to incorporate additional components within our indexing strategy or employ alternative methods like creating multiple indexes that cater specifically towards different sorting scenarios. PostgreSQL provides a solution for this through operator families, wherein we can define the desired ordering characteristics of an attribute (or set of attributes) in complex datatypes such as two-dimensional points on Cartesian coordinates when considering order by X or Y values separately — essentially providing us with customizable flexibility to establish our unique data arrangement within a B-tree index. How to Check if My Index Is Used We can always check if the index was used with EXPLAIN ANALYZE. We use it like this: EXPLAIN ANALYZE SELECT … This query returns a textual representation of the plan with all the operations listed. If we see the index scan among them, then the index has been used. There are other tools. For instance, Metis shows which indexes were used: Why Isn’t My Index Used? Let’s now go through the reasons why an index may not be used. Even though we focus on B-tree indexes here, similar issues can apply to other index types (BRIN, GiST, GIN, etc.). The general principle is: that either the database can’t use the index, or the database thinks that the index will make the query slower. All the scenarios we explain below simply come down to these two principles and only manifest themselves differently. The Index Will Make the Query Slower Let’s start with the cases when the database thinks the index will make the query slower. Table Scan Is Cheaper The very first reason is that the table scan may be faster than the index. This can be the case for small tables. A trivial example includes the following. Let’s create a table: CREATE TABLE people (id INT, name VARCHAR, last_name VARCHAR); Let’s insert two rows: INSERT INTO people VALUES (1, 'John', 'Doe'), (2, 'Mary', 'Alice') Let’s now create an index: CREATE INDEX people_idx ON people(id); If we now try to query the table and get the execution plan, we get the following: EXPLAIN ANALYZE SELECT * FROM people WHERE id = 2 Seq Scan on people (cost=0.00..1.02 rows=1 width=36) (actual time=0.014..0.016 rows=1 loops=1) Filter: (id = 2) Rows Removed by Filter: 1 Planning Time: 0.068 ms Execution Time: 0.123 ms We can see the database decided to scan the table instead of using an index. That is because it was cheaper to scan the table. We can disable scans if possible with: set enable_seqscan=off Let’s now try the query: EXPLAIN ANALYZE SELECT * FROM people WHERE id = 2 Index Scan using people_idx on people (cost=0.13..8.14 rows=1 width=68) (actual time=0.064..0.066 rows=1 loops=1) Index Cond: (id = 2) Planning Time: 0.066 ms Execution Time: 0.170 ms We can see that the sequential scan cost was 1.02 whereas the index scan cost was 8.14. Therefore, the database was right to scan the table. The Query Is Not Selective Enough Another reason is that the query may not be selective enough. This is the same case as before, only it manifests itself differently. This time, we don’t deal with small tables, but we extract too many rows from the table. Let’s add more rows to the table: INSERT INTO people ( WITH RECURSIVE numbers(n) AS ( VALUES (3) UNION ALL SELECT n+1 FROM numbers WHERE n < 1000 ) SELECT n, 'Jack ' || n, 'Dean' FROM numbers ) Let’s now query like this: EXPLAIN ANALYZE SELECT * FROM people WHERE id <= 999 Seq Scan on people (cost=0.00..7.17 rows=5 width=68) (actual time=0.011..0.149 rows=999 loops=1) Filter: (id <= 999) Rows Removed by Filter: 1 Planning Time: 0.072 ms Execution Time: 0.347 ms We can see the sequential scan again. Let’s see the cost when we disable scans: Index Scan using people_idx on people (cost=0.28..47.76 rows=999 width=17) (actual time=0.012..0.170 rows=999 loops=1) Index Cond: (id <= 999) Planning Time: 0.220 ms Execution Time: 0.328 ms So we can see the scan is 7.17 versus 47.76 for the index scan. This is because the query extracts nearly all the rows from the table. It’s not very selective. However, if we try to pick just one row, we should get this: EXPLAIN ANALYZE SELECT * FROM people WHERE id = 2 Index Scan using people_idx on people (cost=0.28..8.29 rows=1 width=17) (actual time=0.018..0.020 rows=1 loops=1) Index Cond: (id = 2) Planning Time: 0.064 ms Execution Time: 0.126 ms Keep in mind that partial indexes may be less or more selective and affect the execution plans. LIMIT Clause Misleads the Database The LIMIT clause may mislead the database and make it think that the sequential scan will be faster. Let’s take this query: EXPLAIN ANALYZE SELECT * FROM people WHERE id <= 50 LIMIT 1 We want to find the rows with id less or equal to fifty. However, we take only one row. The plan looks like this: Limit (cost=0.00..0.39 rows=1 width=17) (actual time=0.016..0.017 rows=1 loops=1) -> Seq Scan on people (cost=0.00..19.50 rows=50 width=17) (actual time=0.015..0.015 rows=1 loops=1) Filter: (id <= 50) Planning Time: 0.073 ms Execution Time: 0.131 ms We can see the database decided to scan the table. However, let’s change the LIMIT to three rows: EXPLAIN ANALYZE SELECT * FROM people WHERE id <= 50 LIMIT 3 Limit (cost=0.28..0.81 rows=3 width=17) (actual time=0.018..0.019 rows=3 loops=1) -> Index Scan using people_idx on people (cost=0.28..9.15 rows=50 width=17) (actual time=0.017..0.018 rows=3 loops=1) Index Cond: (id <= 50) Planning Time: 0.073 ms Execution Time: 0.179 ms This time we get the index scan. This clearly shows that the LIMIT clause affects the execution plan. However, notice that both of the queries executed in 131 and 179 milliseconds respectively. If we disable the scans, we get the following: Limit (cost=0.28..0.81 rows=3 width=17) (actual time=0.019..0.021 rows=3 loops=1) -> Index Scan using people_idx on people (cost=0.28..9.15 rows=50 width=17) (actual time=0.018..0.019 rows=3 loops=1) Index Cond: (id <= 50) Planning Time: 0.074 ms Execution Time: 0.090 ms In this case, the table scan was indeed faster than the index scan. This obviously depends on the query we execute. Be careful with using LIMIT. Inaccurate Statistics Mislead the Database As we saw before, the database uses many heuristics to calculate the query cost. The query planner estimates are based on the number of rows that each operation will return. This is based on the table statistics that may be off if we change the table contents significantly or when columns are dependent on each other (with multivariate statistics). As explained in the PostgreSQL documentation, statistics like reltuples and relpages are not updated on the fly. They get updated periodically or when we run commands like VACUUM, ANALYZE, or CREATE INDEX. Always keep your statistics up to date. Run ANALYZE periodically to make sure that your numbers are not off, and always run ANALYZE after batch-loading multiple rows. You can also read more in PostgreSQL documentation about multivariate statistics and how they affect the planner. The Database Is Configured Incorrectly The planner uses various constants to estimate the query cost. These constants should reflect the hardware we use and the infrastructure backing our database. One of the prominent examples is random_page_cost. This constant is meant to represent the cost of accessing a data page randomly. It’s set to 4 by default whereas the seq_page_cost constant is set to 1. This makes a lot of sense for HDDs that are much slower for random access than the sequential scan. However, SSDs and NVMes do not suffer that much with random access. Therefore, we may want to change this constant to a much lower value (like 2 or even 1.1). This heavily depends on your hardware and infrastructure, so do not tweak these values blindly. There Are Better Indexes Another case for not using our index is when there is a better index in place. Let’s create the following index: CREATE INDEX people_idx2 ON people(id) INCLUDE (name); Let’s now run this query: EXPLAIN ANALYZE SELECT id, name FROM people WHERE id = 123 Index Only Scan using people_idx2 on people (cost=0.28..8.29 rows=1 width=12) (actual time=0.026..0.027 rows=1 loops=1) Index Cond: (id = 123) Heap Fetches: 1 Planning Time: 0.208 ms Execution Time: 0.131 ms We can see it uses people_idx2 instead of people_idx. The database could use people_idx as well but people_idx2 covers all the columns and so can be used as well. We can drop people_idx2 to see how it affects the query: Index Scan using people_idx on people (cost=0.28..8.29 rows=1 width=12) (actual time=0.010..0.011 rows=1 loops=1) Index Cond: (id = 123) Planning Time: 0.108 ms Execution Time: 0.123 ms We can see that using people_idx had the same estimated cost but was faster. Therefore, always tune your indexes to match your production queries. The Index Can’t Be Used Let’s now examine cases when the index can’t be used for some technical reasons. The Index Uses Different Sorting Each index must keep the specific order of rows to be able to run the binary search. If we query for rows in a different order, we may not be able to use the index (or we would need to sort all the rows afterward). Let’s see that. Let’s drop all the indexes and create this one: CREATE INDEX people_idx3 ON people(id, name, last_name) Notice that the index covers all the columns in the table and stores them in ascending order for every column. Let’s now run this query: EXPLAIN ANALYZE SELECT id, name FROM people ORDER BY id, name, last_name Index Only Scan using people_idx3 on people (cost=0.15..56.90 rows=850 width=68) (actual time=0.006..0.007 rows=0 loops=1) Heap Fetches: 0 Planning Time: 0.075 ms Execution Time: 0.117 ms We can see we used the index to scan the table. However, let’s now sort the name DESC: EXPLAIN ANALYZE SELECT id, name FROM people ORDER BY id, name DESC, last_name Sort (cost=66.83..69.33 rows=1000 width=17) (actual time=0.160..0.211 rows=1000 loops=1) Sort Key: id, name DESC, last_name Sort Method: quicksort Memory: 87kB -> Seq Scan on people (cost=0.00..17.00 rows=1000 width=17) (actual time=0.009..0.084 rows=1000 loops=1) Planning Time: 0.120 ms Execution Time: 0.427 ms We can see the index wasn’t used. That is because the index stores the rows in a different order than the query requested. Always configure indexes accordingly to your queries to avoid mismatches like this one. The Index Stores Different Columns Another example is when the order matches but we don’t query the columns accordingly. Let’s run this query: EXPLAIN ANALYZE SELECT id, name FROM people ORDER BY id, last_name Sort (cost=66.83..69.33 rows=1000 width=17) (actual time=0.157..0.198 rows=1000 loops=1) Sort Key: id, last_name Sort Method: quicksort Memory: 87kB -> Seq Scan on people (cost=0.00..17.00 rows=1000 width=17) (actual time=0.012..0.086 rows=1000 loops=1) Planning Time: 0.081 ms Execution Time: 0.388 ms Notice that the index wasn’t used. That is because the index stores all three columns but we query only two of them. Again, always tune your indexes to store the data you need. The Query Uses Functions Differently Let’s now run this query with a function: EXPLAIN ANALYZE SELECT id, name FROM people WHERE abs(id) = 123 Seq Scan on people (cost=0.00..22.00 rows=5 width=12) (actual time=0.019..0.087 rows=1 loops=1) Filter: (abs(id) = 123) Rows Removed by Filter: 999 Planning Time: 0.124 ms Execution Time: 0.181 ms We can see the index wasn’t used. That is because the index was created on a raw value of the id column, but the query tries to get abs(id). The engine could do some more extensive analysis to understand that the function doesn’t change anything in this case, but it decided not to. To make the query faster, we can either not use the function in the query (recommended) or create an index for this query specifically: CREATE INDEX people_idx4 ON people(abs(id)) And then we get: Bitmap Heap Scan on people (cost=4.31..11.32 rows=5 width=12) (actual time=0.022..0.024 rows=1 loops=1) Recheck Cond: (abs(id) = 123) Heap Blocks: exact=1 -> Bitmap Index Scan on people_idx4 (cost=0.00..4.31 rows=5 width=0) (actual time=0.016..0.017 rows=1 loops=1) Index Cond: (abs(id) = 123) Planning Time: 0.209 ms Execution Time: 0.159 ms The Query Uses Different Data Types Yet another example is when we store values with a different data type in the index. Let’s run this query: EXPLAIN ANALYZE SELECT id, name FROM people WHERE id = 123::numeric Seq Scan on people (cost=0.00..22.00 rows=5 width=12) (actual time=0.030..0.156 rows=1 loops=1) Filter: ((id)::numeric = '123'::numeric) Rows Removed by Filter: 999 Planning Time: 0.093 ms Execution Time: 0.278 ms Even though 123 and 123::numeric represent the same value, we can’t use the index because it stores integers instead of numeric types. To fix the issue, we can create a new index targeting this query or change the casting to match the data type: EXPLAIN ANALYZE SELECT id, name FROM people WHERE id = 123::int Bitmap Heap Scan on people (cost=4.18..12.64 rows=4 width=36) (actual time=0.005..0.006 rows=0 loops=1) Recheck Cond: (id = 123) -> Bitmap Index Scan on people_idx3 (cost=0.00..4.18 rows=4 width=0) (actual time=0.004..0.004 rows=0 loops=1) Index Cond: (id = 123) Planning Time: 0.078 ms Execution Time: 0.088 ms Operators Are Not Supported Yet another example of when an index can’t be used is when we query data with an unsupported operator. Let’s create such an index: CREATE INDEX people_idx5 ON people(name) Let’s now query the table with the following; EXPLAIN ANALYZE SELECT name FROM people WHERE name = 'Jack 123' Index Only Scan using people_idx5 on people (cost=0.28..8.29 rows=1 width=8) (actual time=0.025..0.026 rows=1 loops=1) Index Cond: (name = 'Jack 123'::text) Heap Fetches: 1 Planning Time: 0.078 ms Execution Time: 0.139 ms We can see the index worked. However, let’s now change the operator to ILIKE: EXPLAIN ANALYZE SELECT name FROM people WHERE name ILIKE 'Jack 123' Seq Scan on people (cost=0.00..19.50 rows=1 width=8) (actual time=0.075..0.610 rows=1 loops=1) Filter: ((name)::text ~~* 'Jack 123'::text) Rows Removed by Filter: 999 Planning Time: 0.130 ms Execution Time: 0.725 ms We can see the database decided not to use an index. This is because we use the ILIKE operator which is not supported with a regular B-tree index. Therefore, always use the appropriate operators to use indexes efficiently. Keep in mind that operators can be prone to various settings. For instance different collation may cause an index to be ignored. Testing Indexes With HypoPG To test various indexes, we don’t need to create them. We can use the HypoPG extension to analyze the index without creating it in the database. Let’s see that in action. Drop all the indexes and run the following query: EXPLAIN ANALYZE SELECT * FROM people WHERE id = 2 Seq Scan on people (cost=0.00..19.50 rows=1 width=17) (actual time=0.011..0.079 rows=1 loops=1) Filter: (id = 2) Rows Removed by Filter: 999 Planning Time: 0.103 ms Execution Time: 0.163 ms We can see that no index was used as there were no indexes at all. We can now see what would happen if we had an index. Let’s first pretend as we created it: SELECT * FROM hypopg_create_index('CREATE INDEX ON people (id)'); 13563 <13563>btree_people_id And let’s now ask the database if it would be used (notice there is no ANALYZE): EXPLAIN SELECT * FROM people WHERE id = 2 Index Scan using "<13563>btree_people_id" on people (cost=0.00..8.01 rows=1 width=17) This way we can test various indexes. Summary Indexes may be tricky and there are many reasons why they are not used. However, ultimately it all goes down to either the database not being able to use the index or thinking that it would make things slower. Fortunately, we can easily verify if indexes are used with EXPLAIN ANALYZE, and we can also test new indexes with HypoPG.

Are you a software developer or other tech professional? If you’re reading this, chances are pretty good that the answer is "yes." Long story short — we want DZone to work for you! We're asking that you take our annual community survey so we can better serve you! ^^ You can also enter the drawing for a chance to receive an exclusive DZone Swag Pack! The software development world moves fast, and we want to keep up! Across our community, we found that readers come to DZone for various reasons, including to learn about new development trends and technologies, find answers to help solve problems they have, connect with other peers, publish their content, and expand their personal brand's audience. In order to continue helping the DZone Community reach goals such as these, we need to know more about you, your learning preferences, and your overall experience on dzone.com and with the DZone team. For this year's DZone Community research, our primary goals are to: Learn about developer tech preferences and habits Identify content types and topics that developers want to get more information on Share this data for public consumption! To support our Community research, we're focusing on several primary areas in the survey: You, including your experience, the types of software you work on, and the tools you use How you prefer to learn and what you want to learn more about on dzone.com The ways in which you engage with DZone, your content likes vs. dislikes, and your overall journey on dzone.com As a community-driven site, our relationships with our members and contributors is invaluable, and we want to make sure that we continue to serve our audience to the best of our ability. If you're curious to see the report from the 2023 Community survey, feel free to check it out here! Thank you in advance for your participation!—Your favorite DZone Content and Community team

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. Kubernetes is driving the future of cloud computing, but its security challenges require us to adopt a full-scale approach to ensure the safety of our environments. Security is not a one-size-fits-all solution; security is a spectrum, influenced by the specific context in which it is applied. Security professionals in the field rarely declare anything as entirely secure, but always as more or less secure than alternatives. In this article, we are going to present various methods to brace the security of your containers. Understanding and Mitigating Container Security Threats To keep your containerized systems secure, it's important to understand the threats they face. Just like a small leak can sink a ship, even a tiny vulnerability can cause big issues. This section will help you gain a deeper understanding of container security and will provide guidance on how to mitigate the threats that come with it. Core Principles of Container Security Attackers often target containers to hijack their compute power — a common example is to gain access for unauthorized cryptocurrency mining. Beyond this, a compromised container can expose sensitive data, including customer information and workload details. In more advanced attacks, the goal is to escape the container and infiltrate the underlying node. If the attacker succeeds, they can move laterally across the cluster, gaining ongoing access to critical resources such as user code, processing power, and valuable data across other nodes. One particularly dangerous attack method is container escape, where an attacker leverages the fact that containers share the host's kernel. If they gain elevated privileges within a compromised container, they could potentially access data or processes in other containers on the same host. Additionally, the Kubernetes control plane is a prime target. If an attacker compromises one of the control plane components, they can manipulate the entire environment, potentially taking it offline or causing significant disruption. Furthermore, if the etcd database is compromised, attackers could alter or destroy the cluster, steal secrets and credentials, or gather enough information to replicate the application elsewhere. Defense in Depth Maintaining a secure container environment requires a layered strategy that underscores the principle of defense in depth. This approach involves implementing multiple security controls at various levels. By deploying overlapping security measures, you create a system where each layer of defense reinforces the others. This way, even if one security measure is breached, the others continue to protect the environment. Figure 1. Defense-in-depth strategy Understanding the Attack Surface Part of the security strategy is understanding and managing the attack surface, which encompasses all potential points of exploitation, including container images, runtime, orchestration tools, the host, and network interfaces. Reducing the attack surface means simplifying the system and minimizing unnecessary components, services, and code. By limiting what is running and enforcing strict access controls, you decrease the opportunities for vulnerabilities to exist or be exploited, making the system more secure and harder for attackers to penetrate. Common Threats and Mitigation Strategies Let's shift our focus to the everyday threats in container security and discover the tools you can immediately put to work to safeguard your systems. Vulnerable Container Images Relying on container images with security vulnerabilities poses significant risks as these vulnerable images often include outdated software or components with publicly known vulnerabilities. A vulnerability, in this context, is essentially a flaw in the code that malicious actors can leverage to trigger harmful outcomes. An example of this is the infamous Heartbleed flaw in the OpenSSL library, which allowed attackers to access sensitive data by exploiting a coding error. When such flaws are present in container images, they create opportunities for attackers to breach systems, leading to potential data theft or service interruptions. Best practices to secure container images include the following: To effectively reduce the attack surface, start by using minimal base imagesthat include only the essential components required for your application. This approach minimizes potential vulnerabilities and limits what an attacker can exploit. Tools like Docker's FROM scratch or distroless images can help create these minimal environments. Understanding and managing container image layers is crucial as each layer can introduce vulnerabilities. By keeping layers minimal and only including what is necessary, you reduce potential attack vectors. Use multi-stage builds to keep the final image lean and regularly review and update your Dockerfiles to remove unnecessary layers. It's important to avoid using unverified or outdated images. Unverified images from public repositories may contain malware, backdoors, or other malicious components. Outdated images often have unpatched vulnerabilities that attackers can exploit. To mitigate these risks, always source images from trusted repositories and regularly update them to the latest versions. Insecure Container Runtime An insecure container runtime is a critical threat as it can lead to privilege escalation, allowing attackers to gain elevated access within the system. With elevated access, attackers can disrupt services by modifying or terminating critical processes, causing downtime and impacting the availability of essential applications. They can gain full control over the container environment, manipulating configurations to deploy malicious containers or introduce malware, which can be used as a launchpad for further attacks. Best practices for hardening the container runtime include the following: Implementing strict security boundaries and adhering to the principle of least privilege are essential for protecting the container runtime. Containers should be configured to run with only the permissions they need to function, minimizing the potential impact of a security breach. This involves setting up role-based access controls. Admission control is a critical aspect of runtime security that involves validating and regulating requests to create or update containers in the cluster. By employing admission controllers, you can enforce security policies and ensure that only compliant and secure container configurations are deployed. This can include checking for the use of approved base images, ensuring that security policies are applied, and verifying that containers are not running as root. Tools like Open Policy Agent (OPA) can be integrated into your Kubernetes environment to provide flexible and powerful admission control capabilities. Here's an example for OPA policy that acts as a gatekeeper, ensuring no container runs with root privileges: Shell package kubernetes.admission deny[msg] { input.request.kind.kind == "Pod" input.request.object.spec.containers[_].securityContext.runAsUser == 0 msg = "Containers must not run as root." } There are a few practices to avoid when securing container runtime: If a container running as root is compromised, an attacker can gain root-level access to the host system, potentially leading to a full system takeover. When containers have unrestricted access to host resources, like the file system, network, or devices, a compromised container could exploit this access to then tamper with the host system, steal sensitive data, or disrupt other services. To prevent such scenarios, use tools like seccomp and AppArmor. These tools can restrict the system calls that containers make and enforce specific security policies. By applying these controls, you can confine containers to their intended operations, protecting the host system from potential breaches or unauthorized activities. Misconfigured Kubernetes Settings Misconfigured Kubernetes settings are a significant threat as they expose the cluster to attacks through overly permissive network policies, weak access controls, and poor secrets management: Overly permissive network policies enable attackers to intercept and tamper with data. Weak access controls allow unauthorized users to perform administrative tasks, disrupt services, and alter configurations. Poor secrets management exposes sensitive information like API keys and passwords, enabling attackers to escalate privileges. Best practices for secure Kubernetes configuration are as follows: The risk of transmitting sensitive information without protection is that it can be intercepted or tampered with by malicious actors during transit. To mitigate this risk, secure all communication channels with transport layer security (TLS). Kubernetes offers tools like cert-manager to automate the management and renewal of TLS certificates. This ensures that communication between services remains encrypted and secure, thereby protecting your data from interception or manipulation. Network policies control the traffic flow between Pods and services in a Kubernetes cluster. By defining network policies, you can isolate sensitive workloads and reduce the risk of lateral movement in case of a compromise. Use Kubernetes' native NetworkPolicy resource to create rules that enforce your desired network security posture. On the other hand, it's important to avoid exposing unnecessary application ports. Exposure of ports provides multiple entry points for attackers, making the cluster more vulnerable to exploits. CI/CD Security CI/CD pipelines are granted extensive permissions, ensuring they can interact closely with production systems and manage updates. However, this extensive access also makes CI/CD pipelines a significant security risk. If compromised, attackers can exploit these broad permissions to manipulate deployments, introduce malicious code, gain unauthorized access to critical systems, steal sensitive data, or create backdoors for ongoing access. There are several best practices to implement when securing CI/CD. The first best practice is ensuring that once a container image is built and deployed, it is immutable. We always want to make sure the Pod is running on exactly what we intended. It also helps in quickly identifying and rolling back to previous stable versions if a security issue arises, maintaining a reliable and predictable deployment process. Implementing immutable deployments involves several key steps to ensure consistency and security: Assign unique version tags to each container image build, avoiding mutable tags like "latest," and use Infrastructure-as-Code tools like Terraform or Ansible to maintain consistent setups. Configure containers with read-only file systems to prevent changes post-deployment. Implement continuous monitoring with tools like Prometheus and runtime security with Falco to help detect and alert to unauthorized changes, maintaining the security and reliability of your deployments. Another best practice is implementing image vulnerability scanning in CI/CD. Vulnerability scanners meticulously analyze the components of container images, identifying known security flaws that could be exploited. Beyond just examining packages managed by tools like DNF or apt, advanced scanners also inspect additional files added during the build process, such as those introduced through Dockerfile commands like ADD, COPY, or RUN. It's important to include both third-party and internally created images in these scans as new vulnerabilities are constantly emerging. To guarantee that images are thoroughly scanned for vulnerabilities before deployment, scanning tools like Clair or Trivy can be directly embedded into your CI/CD pipeline. Do not store sensitive information directly in the source code (e.g., API keys, passwords) as this increases the risk of unauthorized access and data breaches. Use secrets management tools like SOPS, AWS Secrets Manager, or Google Cloud Secret Manager to securely handle and encrypt sensitive information. Conclusion Regularly assessing and improving Kubernetes security measures is not just important — it's essential. By implementing the strategies we introduced above, organizations can protect their Kubernetes environments, ensuring that containerized applications are more secure and resilient against challenges. In the future, we anticipate that attackers will develop more sophisticated methods to specifically bypass Kubernetes' built-in security features. As organizations increasingly rely on Kubernetes for critical workloads, attackers will likely invest time in uncovering new vulnerabilities or weaknesses in Kubernetes' security architecture, potentially leading to breaches that are more difficult to detect and mitigate. The path to a secure Kubernetes environment is clear, and the time to act is now. Prioritize security to safeguard your future. This is an excerpt from DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC.Read the Free Report

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. In recent years, observability has re-emerged as a critical aspect of DevOps and software engineering in general, driven by the growing complexity and scale of modern, cloud-native applications. The transition toward microservices architecture as well as complex cloud deployments — ranging from multi-region to multi-cloud, or even hybrid-cloud, environments — has highlighted the shortcomings of traditional methods of monitoring. In response, the industry has standardized utilizing logs, metrics, and traces as the three pillars of observability to provide a more comprehensive sense of how the application and the entire stack is performing. We now have a plethora of tools to collect, store, and analyze various signals to diagnose issues, optimize performance, and respond to issues. Yet anyone working with Kubernetes will still say that observability in Kubernetes remains challenging. Part of it comes from the inherent complexity of working with Kubernetes, but the fact of the matter is that logs, metrics, and traces alone don't make up observability. Also, the vast ecosystem of observability tooling does not necessarily equate to ease of use or high ROI, especially given today's renewed focus on cost. In this article, we'll dive into some considerations for Kubernetes observability, challenges of and some potential solutions for implementing it, and the oft forgotten aspect of developer experience in observability. Considerations for Kubernetes Observability When considering observability for Kubernetes, most have a tendency to dive straight into tool choices, but it's advisable to take a hard look at what falls under the scope of things to "observe" for your use case. Within Kubernetes alone, we already need to consider: Cluster components – API server, etcd, controller manager, scheduler Node components – kublect, kube-proxy, container runtime Other resources – CoreDNS, storage plugins, Ingress controllers Network – CNI, service mesh Security and access – audit logs, security policies Application – both internal and third-party applications And most often, we inevitably have components that run outside of Kubernetes but interface with many applications running inside. Most notably, we have databases ranging from managed cloud offerings to external data lakes. We also have things like serverless functions, queues, or other Kubernetes clusters that we need to think about. Next, we need to identify the users of Kubernetes as well as the consumers of these observability tools. It's important to consider these personas as building for an internal-only cluster vs. a multi-tenant SaaS cluster may have different requirements (e.g., privacy, compliance). Also, depending on the team composition, the primary consumers of these tools may be developers or dedicated DevOps/SRE teams who will have different levels of expertise with not only these tools but with Kubernetes itself. Only after considering the above factors can we start to talk about what tools to use. For example, if most applications are already on Kubernetes, using a Kubernetes-focused tool may suffice, whereas organizations with lots of legacy components may elect to reuse an existing observability stack. Also, a large organization with various teams mostly operating as independent verticals may opt to use their own tooling stack, whereas a smaller startup may opt to pay for an enterprise offering to simplify the setup across teams. Challenges and Recommendations for Observability Implementation After considering the scope and the intended audience of our observability stack, we're ready to narrow down the tool choices. Largely speaking, there are two options for implementing an observability stack: open source and commercial/SaaS. Open-Source Observability Stack The primary challenge with implementing a fully open-source observability solution is that there is no single tool that covers all aspects. Instead, what we have are ecosystems or stacks of tools that cover different aspects of observability. One of the more popular tech stacks from Prometheus and Grafana Lab's suite of products include: Prometheus for scraping metrics and alerting Loki for collecting logs Tempo for distributed tracing Grafana for visualization While the above setup does cover a vast majority of observability requirements, they still operate as individual microservices and do not provide the same level of uniformity as a commercial or SaaS product. But in recent years, there has been a strong push to at least standardize on OpenTelemetry conventions to unify how to collect metrics, logs, and traces. Since OpenTelemetry is a framework that is tool agnostic, it can be used with many popular open-source tools like Prometheus and Jaeger. Ideally, architecting with OpenTelemetry in mind will make standardization of how to generate, collect, and manage telemetry data easier with the growing list of compliant open-source tools. However, in practice, most organizations will already have established tools or in-house versions of them — whether that is the EFK (Elasticsearch, Fluentd, Kibana) stack or Prometheus/Grafana. Instead of forcing a new framework or tool, apply the ethos of standardization and improve what and how telemetry data is collected and stored. Finally, one of the common challenges with open-source tooling is dealing with storage. Some tools like Prometheus cannot scale without offloading storage with another solution like Thanos or Mimir. But in general, it's easy to forget to monitor the observability tooling health itself and scale the back end accordingly. More telemetry data does not necessarily equal more signals, so keep a close eye on the volume and optimize as needed. Commercial Observability Stack On the commercial offering side, we usually have agent-based solutions where telemetry data is collected from agents running as DaemonSets on Kubernetes. Nowadays, almost all commercial offerings have a comprehensive suite of tools that combine into a seamless experience to connect logs to metrics to traces in a single user interface. The primary challenge with commercial tools is controlling cost. This usually comes in the form of exposing cardinality from tags and metadata. In the context of Kubernetes, every Pod has tons of metadata related to not only Kubernetes state but the state of the associated tooling as well (e.g., annotations used by Helm or ArgoCD). These metadata then get ingested as additional tags and date fields by the agents. Since commercial tools have to index all the data to make telemetry queryable and sortable, increased cardinality from additional dimensions (usually in the form of tags) causes issues with performance and storage. This directly results in higher cost to the end user. Fortunately, most tools now allow the user to control which tags to index and even downsample data to avoid getting charged for repetitive data points that are not useful. Be aggressive with filters and pipeline logic to only index what is needed; otherwise, don't be surprised by the ballooning bill. Remembering the Developer Experience Regardless of the tool choice, one common pitfall that many teams face is over-optimizing for ops usage and neglecting the developer experience when it comes to observability. Despite the promise of DevOps, observability often falls under the realm of ops teams, whether that be platform, SRE, or DevOps engineering. This makes it easy for teams to build for what they know and what they need, over-indexing on infrastructure and not investing as much on application-level telemetry. This ends up alienating developers to invest less time or become too reliant on their ops counterparts for setup or debugging. To make observability truly useful for everyone involved, don't forget about these points: Access. It's usually more of a problem with open-source tools, but make sure access to logs, dashboards, and alerts are not gated by unnecessary approvals. Ideally, having quick links from existing mediums like IDEs or Slack can make tooling more accessible. Onboarding. It's rare for developers to go through the same level of onboarding in learning how to use any of these tools. Invest some time to get them up to speed. Standardization vs. flexibility. While a standard format like JSON is great for indexing, it may not be as human readable and is filled with extra information. Think of ways to present information in a usable format. At the end of the day, the goals of developers and ops teams should be aligned. We want tools that are easy to integrate, with minimal overhead, that produce intuitive dashboards and actionable, contextual information without too much noise. Even with the best tools, you still need to work with developers who are responsible for generating telemetry and also acting on it, so don't neglect the developer experience entirely. Final Thoughts Observability has been a hot topic in recent years due to several key factors, including the rise of complex, modern software coupled with DevOps and SRE practices to deal with that complexity. The community has moved past the simple notion of monitoring to defining the three pillars of observability as well as creating new frameworks to help with generation, collection, and management of these telemetry data. Observability in a Kubernetes context has remained challenging so far given the large scope of things to "observe" as well as the complexity of each component. With the open source ecosystem, we have seen a large fragmentation of specialized tools that is just now integrating into a standard framework. On the commercial side, we have great support for Kubernetes, but cost control has been a huge issue. And to top it off, lost in all of this complexity is the developer experience in helping feed data into and using the insights from the observability stack. But as the community has done before, tools and experience will continue to improve. We already see significant research and advances in how AI technology can improve observability tooling and experience. Not only do we see better data-driven decision making, but generative AI technology can also help surface information better in context to make tools more useful without too much overhead. This is an excerpt from DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC.Read the Free Report

By Yitaek Hwang CORE

Kong Gateway is an open-source API gateway that ensures only the right requests get in while managing security, rate limiting, logging, and more. OPA (Open Policy Agent) is an open-source policy engine that takes control of your security and access decisions. Think of it as the mind that decouples policy enforcement from your app, so your services don’t need to stress about enforcing rules. Instead, OPA does the thinking with its Rego language, evaluating policies across APIs, microservices, or even Kubernetes. It’s flexible, and secure, and makes updating policies a breeze. OPA works by evaluating three key things: input (real-time data like requests), data (external info like user roles), and policy (the logic in Rego that decides whether to "allow" or "deny"). Together, these components allow OPA to keep your security game strong while keeping things simple and consistent. What Are We Seeking to Accomplish or Resolve? Oftentimes, the data in OPA is like a steady old friend — static or slowly changing. It’s used alongside the ever-changing input data to make smart decisions. But, imagine a system with a sprawling web of microservices, tons of users, and a massive database like PostgreSQL. This system handles a high volume of transactions every second and needs to keep up its speed and throughput without breaking a sweat. Fine-grained access control in such a system is tricky, but with OPA, you can offload the heavy lifting from your microservices and handle it at the gateway level. By teaming up with Kong API Gateway and OPA, you get both top-notch throughput and precise access control. How do you maintain accurate user data without slowing things down? Constantly hitting that PostgreSQL database to fetch millions of records is both expensive and slow. Achieving both accuracy and speed usually requires compromises between the two. Let’s aim to strike a practical balance by developing a custom plugin (at the gateway level) that frequently loads and locally caches data for OPA to use in evaluating its policies. Demo For the demo, I’ve set up sample data in PostgreSQL, containing user information such as name, email, and role. When a user tries to access a service via a specific URL, OPA evaluates whether the request is permitted. The Rego policy checks the request URL (resource), method, and the user’s role, then returns either true or false based on the rules. If true, the request is allowed to pass through; if false, access is denied. So far, it's a straightforward setup. Let’s dive into the custom plugin. For a clearer understanding of its implementation, please refer to the diagram below. When a request comes through the Kong Proxy, the Kong custom plugin would get triggered. The plugin would fetch the required data and pass it to OPA along with the input/query. This data fetch has two parts to it: one would be to look up Redis to find the required values, and if found, pass it along to OPA; if else, it would further query the Postgres and fetch the data and cache it in Redis before passing it along to OPA. We can revisit this when we run the commands in the next section and observe the logs. OPA makes a decision (based on the policy, input, and data) and if it's allowed, Kong will proceed to send that request to the API. Using this approach, the number of queries to Postgres is significantly reduced, yet the data available for OPA is fairly accurate while preserving the low latency. To start building a custom plugin, we need a handler.lua where the core logic of the plugin is implemented and a schema.lua which, as the name indicates, defines the schema for the plugin’s configuration. If you are starting to learn how to write custom plugins for Kong, please refer to this link for more info. The documentation also explains how to package and install the plugin. Let’s proceed and understand the logic of this plugin. The first step of the demo would be to install OPA, Kong, Postgres, and Redis on your local setup or any cloud setup. Please clone into this repository. Review the docker-compose yaml which has the configurations defined to deploy all four services above. Observe the Kong Env variables to see how the custom plugin is loaded. Run the below commands to deploy the services: Dockerfile docker-compose build docker-compose up Once we verify the containers are up and running, Kong manager and OPA are available on respective endpoints https://localhost:8002 and https://localhost:8181 as shown below: Create a test service, route and add our custom kong plugin to this route by using the below command: Shell curl -X POST http://localhost:8001/config -F config=@config.yaml The OPA policy, defined in authopa.rego file, is published and updated to the OPA service using the below command: Shell curl -X PUT http://localhost:8181/v1/policies/mypolicyId -H "Content-Type: application/json" --data-binary @authopa.rego This sample policy grants access to user requests only if the user is accessing the /demo path with a GET method and has the role of "Moderator". Additional rules can be added as needed to tailor access control based on different criteria. JSON opa_policy = [ { "path": "/demo", "method": "GET", "allowed_roles": ["Moderator"] } ] Now the setup is ready, but before testing, we need some test data to add in Postgres. I added some sample data (name, email, and role) for a few employees as shown below (please refer to the PostgresReadme). Here’s a sample failed and successful request: Now, to test the core functionality of this custom plugin, let’s make two consecutive requests and check the logs for how the data retrieval is happening. Here are the logs: JSON 2024/09/13 14:05:05 [error] 2535#0: *10309 [kong] redis.lua:19 [authopa] No data found in Redis for key: alice@example.com, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] handler.lua:25 [authopa] Fetching roles from PostgreSQL for email: alice@example.com, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] postgres.lua:43 [authopa] Fetched roles: Moderator, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] handler.lua:29 [authopa] Caching user roles in Redis, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] redis.lua:46 [authopa] Data successfully cached in Redis, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 [kong] opa.lua:37 [authopa] Is Allowed by OPA: true, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "ebbb8b5b57ff4601ff194907e35a3002" 2024/09/13 14:05:05 [info] 2535#0: *10309 client 192.168.96.1 closed keepalive connection ------------------------------------------------------------------------------------------------------------------------ 2024/09/13 14:05:07 [info] 2535#0: *10320 [kong] redis.lua:23 [authopa] Redis result: {"roles":["Moderator"],"email":"alice@example.com"}, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "75bf7a4dbe686d0f95e14621b89aba12" 2024/09/13 14:05:07 [info] 2535#0: *10320 [kong] opa.lua:37 [authopa] Is Allowed by OPA: true, client: 192.168.96.1, server: kong, request: "GET /demo HTTP/1.1", host: "localhost:8000", request_id: "75bf7a4dbe686d0f95e14621b89aba12" The logs show that for the first request when there’s no data in Redis, the data is being fetched from Postgres and cached in Redis before sending it forward to OPA for evaluation. In the subsequent request, since the data is available in Redis, the response would be much faster. Conclusion In conclusion, by combining Kong Gateway with OPA and implementing the custom plugin with Redis caching, we effectively balance accuracy and speed for access control in high-throughput environments. The plugin minimizes the number of costly Postgres queries by caching user roles in Redis after the initial query. On subsequent requests, the data is retrieved from Redis, significantly reducing latency while maintaining accurate and up-to-date user information for OPA policy evaluations. This approach ensures that fine-grained access control is handled efficiently at the gateway level without sacrificing performance or security, making it an ideal solution for scaling microservices while enforcing precise access policies.

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. In the past, before CI/CD and Kubernetes came along, deploying software to Kubernetes was a real headache. Developers would build stuff on their own machines, then package it and pass it to the operations team to deploy it on production. This approach would frequently lead to delays, miscommunications, and inconsistencies between environments. Operations teams had to set up the deployments themselves, which increased the risk of human errors and configuration issues. When things went wrong, rollbacks were time consuming and disruptive. Also, without automated feedback and central monitoring, it was tough to keep an eye on how builds and deployments were progressing or to identify production issues. With the advent of CI/CD pipelines combined with Kubernetes, deploying software is smoother. Developers can simply push their code, which triggers builds, tests, and deployments. This enables organizations to ship new features and updates more frequently and reduce the risk of errors in production. This article explains the CI/CD transformation with Kubernetes and provides a step-by-step guide to building a pipeline. Why CI/CD Should Be Combined With Kubernetes CI/CD paired with Kubernetes is a powerful combination that makes the whole software development process smoother. Kubernetes, also known as K8s, is an open-source system for automating the deployment, scaling, and management of containerized applications. CI/CD pipelines, on the other hand, automate how we build, test, and roll out software. When you put them together, you can deploy more often and faster, boost software quality with automatic tests and checks, cut down on the chance of pushing out buggy code, and get more done by automating tasks that used to be done by hand. CI/CD with Kubernetes helps developers and operations teams work better together by giving them a shared space to do their jobs. This teamwork lets companies deliver high-quality applications rapidly and reliably, gaining an edge in today's fast-paced world. Figure 1 lays out the various steps: Figure 1. Push-based CI/CD pipeline with Kubernetes and monitoring tools There are several benefits in using CI/CD with Kubernetes, including: Faster and more frequent application deployments, which help in rolling out new features or critical bug fixes to the users Improved quality by automating testing and incorporating quality checks, which helps in reducing the number of bugs in your applications Reduced risk of deploying broken code to production since CI/CD pipelines can conduct automated tests and roll-back deployments if any problems exist Increased productivity by automating manual tasks, which can free developers' time to focus on important projects Improved collaboration between development and operations teams since CI/CD pipelines provide a shared platform for both teams to work Tech Stack Options There are different options available if you are considering building a CI/CD pipeline with Kubernetes. Some of the popular ones include: Open-source tools such as Jenkins, Argo CD, Tekton, Spinnaker, or GitHub Actions Enterprise tools, including but not limited to, Azure DevOps, GitLab CI/CD, or AWS CodePipeline Deciding whether to choose an open-source or enterprise platform to build efficient and reliable CI/CD pipelines with Kubernetes will depend on your project requirements, team capabilities, and budget. Impact of Platform Engineering on CI/CD With Kubernetes Platform engineering builds and maintains the underlying infrastructure and tools (the "platform") that development teams use to create and deploy applications. When it comes to CI/CD with Kubernetes, platform engineering has a big impact on making the development process better. It does so by hiding the complex parts of the underlying infrastructure and giving developers self-service options. Platform engineers manage and curate tools and technologies that work well with Kubernetes to create a smooth development workflow. They create and maintain CI/CD templates that developers can reuse, allowing them to set up pipelines without thinking about the details of the infrastructure. They also set up rules and best practices for containerization, deployment strategies, and security measures, which help maintain consistency and reliability across different applications. What's more, platform engineers provide ways to observe and monitor applications running in Kubernetes, which let developers find and fix problems and make improvements based on data. By building a strong platform, platform engineering helps dev teams zero in on creating and rolling out features more without getting bogged down by the complexities of the underlying tech. It brings together developers, operations, and security teams, which leads to better teamwork and faster progress in how things are built. How to Build a CI/CD Pipeline With Kubernetes Regardless of the tech stack you select, you will often find similar workflow patterns and steps. In this section, I will focus on building a CI/CD pipeline with Kubernetes using GitHub Actions. Step 1: Setup and prerequisites GitHub account – needed to host your code and manage the CI/CD pipeline using GitHub Actions Kubernetes cluster – create one locally (e.g., MiniKube) or use a managed service from Amazon or Azure kubectl – Kubernetes command line tool to connect to your cluster Container registry – needed for storing Docker images; you can either use a cloud provider's registry (e.g., Amazon ECR, Azure Container Registry, Google Artifact Registry) or set up your own private registry Node.js and npm – install Node.js and npm to run the sample Node.js web application Visual Studio/Visual Studio Code – IDE platform for making code changes and submitting them to a GitHub repository Step 2: Create a Node.js web application Using Visual Studio, create a simple Node.js application with a default template. If you look inside, the server.js in-built generated file will look like this: Shell // server.js 'use strict'; var http = require('http'); var port = process.env.PORT || 1337; http.createServer(function (req, res) { res.writeHead(200, { 'Content-Type': 'text/plain' }); res.end('Hello from kubernetes\n'); }).listen(port); Step 3: Create a package.json file to manage dependencies Inside the project, add a new file Package.json to manage dependencies: Shell // Package.Json { "name": "nodejs-web-app1", "version": "0.0.0", "description": "NodejsWebApp", "main": "server.js", "author": { "name": "Sunny" }, "scripts": { "start": "node server.js", "test": "echo \"Running tests...\" && exit 0" }, "devDependencies": { "eslint": "^8.21.0" }, "eslintConfig": { } } Step 4: Build a container image Create a Dockerfile to define how to build your application's Docker image: Shell // Dockerfile # Use the official Node.js image from the Docker Hub FROM node:14 # Create and change to the app directory WORKDIR /usr/src/app # Copy package.json and package-lock.json COPY package*.json ./ # Install dependencies RUN npm install # Copy the rest of the application code COPY . . # Expose the port the app runs on EXPOSE 3000 # Command to run the application CMD ["node", "app.js"] Step 5: Create a Kubernetes Deployment manifest Create a deployment.yaml file to define how your application will be deployed in Kubernetes: Shell // deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nodejs-deployment spec: replicas: 3 selector: matchLabels: app: nodejs-app template: metadata: labels: app: nodejs-app spec: containers: - name: nodejs-container image: nodejswebapp ports: - containerPort: 3000 env: - name: NODE_ENV value: "production" --- apiVersion: v1 kind: Service metadata: name: nodejs-service spec: selector: app: nodejs-app ports: - protocol: TCP port: 80 targetPort: 3000 type: LoadBalancer Step 6: Push code to GitHub Create a new code repository on GitHub, initialize the repository, commit your code changes, and push it to your GitHub repository: Shell git init git add . git commit -m "Initial commit" git remote add origin "<remote git repo url>" git push -u origin main Step 7: Create a GitHub Actions workflow Inside your GitHub repository, go to the Actions tab. Create a new workflow (e.g., main.yml) in the .github/workflows directory. Inside the GitHub repository settings, create Secrets under actions related to Docker and Kubernetes cluster — these are used in your workflow to authenticate: Shell //main.yml name: CI/CD Pipeline on: push: branches: - main jobs: build: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up Node.js uses: actions/setup-node@v2 with: node-version: '14' - name: Install dependencies run: npm install - name: Run tests run: npm test - name: Build Docker image run: docker build -t <your-docker-image> . - name: Log in to Docker Hub uses: docker/login-action@v1 with: username: ${{ secrets.DOCKER_USERNAME } password: ${{ secrets.DOCKER_PASSWORD } - name: Build and push Docker image uses: docker/build-push-action@v2 with: context: . push: true tags: <your-docker-image-tag> deploy: needs: build runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up kubectl uses: azure/setup-kubectl@v1 with: version: 'latest' - name: Set up Kubeconfig run: echo "${{ secrets.KUBECONFIG }" > $HOME/.kube/config - name: Deploy to Kubernetes run: kubectl apply -f deployment.yaml Step 8: Trigger the pipeline and monitor Modify server.js and push it to the main branch; this triggers the GitHub Actions workflow. Monitor the workflow progress. It installs the dependencies, sets up npm, builds the Docker image and pushes it to the container registry, and deploys the application to Kubernetes. Once the workflow is completed successfully, you can access your application that is running inside the Kubernetes cluster. You can leverage open-source monitoring tools like Prometheus and Grafana for metrics. Deployment Considerations There are a few deployment considerations to keep in mind when developing CI/CD pipelines with Kubernetes to maintain security and make the best use of resources: Scalability Use horizontal pod autoscaling to scale your application's Pods based on how much CPU, memory, or custom metrics are needed. This helps your application work well under varying loads. When using a cloud-based Kubernetes cluster, use the cluster autoscaler to change the number of worker nodes as needed to ensure enough resources are available and no money is wasted on idle resources. Ensure your CI/CD pipeline incorporates pipeline scalability, allowing it to handle varying workloads as per your project needs. Security Scan container images regularly to find security issues. Add tools for image scanning into your CI/CD pipeline to stop deploying insecure code. Implement network policies to limit how Pods and services talk to each other inside a Kubernetes cluster. This cuts down on ways attackers could get in. Set up secrets management using Kubernetes Secrets or external key vaults to secure and manage sensitive info such as API keys and passwords. Use role-based access control to control access to Kubernetes resources and CI/CD pipelines. High availability Through multi-AZ or multi-region deployments, you can set up your Kubernetes cluster in different availability zones or regions to keep it running during outages. Pod disruption budgets help you control how many Pods can be down during planned disruptions (like fixing nodes) or unplanned ones (like when nodes fail). Implement health checks to monitor the health of your pods and automatically restart if any fail to maintain availability. Secrets management Store API keys, certificates, and passwords as Kubernetes Secrets, which are encrypted and added to Pods. You can also consider external secrets management tools like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault if you need dynamic secret generation and auditing. Conclusion Leveraging CI/CD pipelines with Kubernetes has become a must-have approach in today's software development. It revolutionizes the way teams build, test, and deploy apps, leading to more efficiency and reliability. By using automation, teamwork, and the strength of container management, CI/CD with Kubernetes empowers organizations to deliver high-quality software at speed. The growing role of AI and ML will likely have an impact on CI/CD pipelines — such as smarter testing, automated code reviews, and predictive analysis to further enhance the development process. When teams adopt best practices, keep improving their pipelines, and are attentive to new trends, they can get the most out of CI/CD with Kubernetes, thus driving innovation and success. This is an excerpt from DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC.Read the Free Report

By Naga Santhosh Reddy Vootukuri CORE

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC. A decade ago, Google introduced Kubernetes to simplify the management of containerized applications. Since then, it has fundamentally transformed the software development and operations landscape. Today, Kubernetes has seen numerous enhancements and integrations, becoming the de facto standard for container orchestration. This article explores the journey of Kubernetes over the past 10 years, its impact on the software development lifecycle (SDLC) and developers, and the trends and innovations that will shape its next decade. The Evolution of Kubernetes Kubernetes, often referred to as K8s, had its first commit pushed to GitHub on June 6, 2014. About a year later, on July 21, 2015, Kubernetes V1 was released, featuring 14,000 commits from 400 contributors. Simultaneously, the Linux Foundation announced the formation of the Cloud Native Computing Foundation (CNCF) to advance state-of-the-art technologies for building cloud-native applications and services. After that, Google donated Kubernetes to the CNCF, marking a significant milestone in its development. Kubernetes addressed a critical need in the software industry: managing the lifecycle of containerized applications. Before Kubernetes, developers struggled with orchestrating containers, leading to inefficiencies and complexities in deployment processes. Kubernetes brought advanced container management functionality and quickly gained popularity due to its robust capabilities in automating the deployment, scaling, and operations of containers. While early versions of Kubernetes introduced the foundation for container orchestration, the project has since undergone significant improvements. Major updates have introduced sophisticated features such as StatefulSets for managing stateful applications, advanced networking capabilities, and enhanced security measures. The introduction of Custom Resource Definitions (CRDs) and Operators has further extended its functionality, allowing users to manage complex applications and workflows with greater ease. In addition, the community has grown significantly over the past decade. According to the 2023 Project Journey Report, Kubernetes now has over 74,680 contributors, making it the second-largest open-source project in the world after Linux. Over the years, Kubernetes has seen numerous enhancements and integrations, becoming the de facto standard for container orchestration. The active open source community and the extensive ecosystem of tools and projects have made Kubernetes an essential technology for modern software development. It is now the "primary container orchestration tool for 71% of Fortune 100 companies" (Project Journey Report). Kubernetes' Impact on the SDLC and Developers Kubernetes abstracts away the complexities of container orchestration and allows developers to focus on development rather than worry about application deployment and orchestration. The benefits and key impacts on the SDLC and developer workflows include enhanced development and testing, efficient deployment, operational efficiency, improved security, and support for microservices architecture. Enhanced Development and Testing Kubernetes ensures consistency for applications running across testing, development, and production environments, regardless of whether the infrastructure is on-premises, cloud based, or a hybrid setup. This level of consistency, along with the capability to quickly spin up and tear down environments, significantly accelerates development cycles. By promoting portability, Kubernetes also helps enterprises avoid vendor lock-in and refine their cloud strategies, leading to a more flexible and efficient development process. Efficient Deployment Kubernetes automates numerous aspects of application deployment, such as service discovery, load balancing, scaling, and self-healing. This automation reduces manual effort, minimizes human error, and ensures reliable and repeatable deployments, reducing downtime and deployment failures. Operational Efficiency Kubernetes efficiently manages resources by dynamically allocating them based on the application's needs. It ensures operations remain cost effective while maintaining optimal performance and use of computing resources by scheduling containers based on resource requirements and availability. Security Kubernetes enhances security by providing container isolation and managing permissions. Its built-in security features allow developers to build secure applications without deep security expertise. Such built-in features include role-based access control, which ensures that only authorized users can access specific resources and perform certain actions. It also supports secrets management to securely store and manage sensitive information like passwords and API keys. Microservices Architecture Kubernetes has facilitated the adoption of microservices architecture by enabling developers to deploy, manage, and scale individual microservices independently. Each microservice can be packaged into a separate container, providing isolation and ensuring that dependencies are managed within the container. Kubernetes' service discovery and load balancing features enable communication between microservices, while its support for automated scaling and self-healing ensures high availability and resilience. Predictions for the Next Decade After a decade, it has become clear that Kubernetes is now the standard technology for container orchestration that's used by many enterprises. According to the CNCF Annual Survey 2023, the usage of Kubernetes continues to grow, with significant adoption across different industries and use cases. Its reliability and flexibility make it a preferred choice for mission-critical applications, including databases, CI/CD pipelines, and AI and machine learning (ML) workloads. As a result, there is a growing demand for new features and enhancements, as well as simplifying concepts for users. The community is now prioritizing improvements that not only enhance user experiences but also promote the sustainability of the project. Figure 1 illustrates the anticipated future trends in Kubernetes, and below are the trends and innovations expected to shape Kubernetes' future in more detail. Figure 1. Future trends in Kubernetes AI and Machine Learning Kubernetes is increasingly used to orchestrate AI and ML workloads, supporting the deployment and management of complex ML pipelines. This simplifies the integration and scaling of AI applications across various environments. Innovations such as Kubeflow — an open-source platform designed to optimize the deployment, orchestration, and management of ML workflows on Kubernetes — enable data scientists to focus more on model development and less on infrastructure concerns. According to the recent CNCF open-source project velocity report, Kubeflow appeared on the top 30 CNCF project list for the first time in 2023, highlighting its growing importance in the ecosystem. Addressing the resource-intensive demands of AI introduces new challenges that contributors are focusing on, shaping the future of Kubernetes in the realm of AI and ML. The Developer Experience As Kubernetes evolves, its complexity can create challenges for new users. Hence, improving the user experience is crucial moving forward. Tools like Backstage are revolutionizing how developers work with Kubernetes and speeding up the development process. The CNCF's open-source project velocity report also states that "Backstage is addressing a significant pain point around developer experience." Moreover, the importance of platform engineering is increasingly recognized by companies. This emerging trend is expected to grow, with the goal of reducing the learning curve and making it easier for developers to adopt Kubernetes, thereby accelerating the development process and improving productivity. CI/CD and GitOps Kubernetes is revolutionizing continuous integration and continuous deployment (CI/CD) pipelines through the adoption of GitOps practices. GitOps uses Git repositories as the source of truth for declarative infrastructure and applications, enabling automated deployments. Tools like ArgoCD and Flux are being widely adopted to simplify the deployment process, reduce human error, and ensure consistency across environments. Figure 2 shows the integration between a GitOps operator, such as ArgoCD, and Kubernetes for managing deployments. This trend is expected to grow, making CI/CD pipelines more robust and efficient. Figure 2. Kubernetes GitOps Sustainability and Efficiency Cloud computing's carbon footprint now exceeds the airline industry, making sustainability and operational efficiency crucial in Kubernetes deployments. The Kubernetes community is actively developing features to optimize resource usage, reduce energy consumption, and enhance the overall efficiency of Kubernetes clusters. CNCF projects like KEDA (Kubernetes event-driven autoscaling) and Karpenter (just-in-time nodes for any Kubernetes cluster) are at the forefront of this effort. These tools not only contribute to cost savings but also align with global sustainability goals. Hybrid and Multi-Cloud Deployments According to the CNCF Annual Survey 2023, multi-cloud solutions are now the norm: Multi-cloud solutions (hybrid and other cloud combinations) are used by 56% of organizations. Deploying applications across hybrid and multi-cloud environments is one of Kubernetes' most significant advantages. This flexibility enables organizations to avoid vendor lock-in, optimize costs, and enhance resilience by distributing workloads across multiple cloud providers. Future developments in Kubernetes will focus on improving and simplifying management across different cloud platforms, making hybrid and multi-cloud deployments even more efficient. Increased Security Features Security continues to be a top priority for Kubernetes deployments. The community is actively enhancing security features to address vulnerabilities and emerging threats. These efforts include improvements to network policies, stronger identity and access management (IAM), and more advanced encryption mechanisms. For instance, the 2024 CNCF open-source project velocity report highlighted that Keycloak, which joined CNCF last year as an incubating project, is playing a vital role in advancing open-source IAM, backed by a large and active community. Edge Computing Kubernetes is playing a crucial role in the evolution of edge computing. By enabling consistent deployment, monitoring, and management of applications at the edge, Kubernetes significantly reduces latency, enhances real-time processing capabilities, and supports emerging use cases like IoT and 5G. Projects like KubeEdge and K3s are at the forefront of this movement. We can expect further optimizations for lightweight and resource-constrained environments, making Kubernetes even more suitable for edge computing scenarios. Conclusion Kubernetes has revolutionized cloud-native computing, transforming how we develop, deploy, and manage applications. As Kelsey Hightower noted in Google's Kubernetes Podcast, "We are only halfway through its journey, with the next decade expected to see Kubernetes mature to the point where it 'gets out of the way' by doing its job so well that it becomes naturally integrated into the background of our infrastructure." Kubernetes' influence will only grow, shaping the future of technology and empowering organizations to innovate and thrive in an increasingly complex landscape. References: "10 Years of Kubernetes" by Bob Killen et al, 2024 CNCF Annual Survey 2023 by CNCF, 2023 "As we reach mid-year 2024, a look at CNCF, Linux Foundation, and top 30 open source project velocity" by Chris Aniszczyk, CNCF, 2024 "Orchestration Celebration: 10 Years of Kubernetes" by Adrian Bridgwater, 2024 "Kubernetes: Beyond Container Orchestration" by Pratik Prakash, 2022 "The Staggering Ecological Impacts of Computation and the Cloud" by Steven Gonzalez Monserrate, 2022 This is an excerpt from DZone's 2024 Trend Report, Kubernetes in the Enterprise: Once Decade-Defining, Now Forging a Future in the SDLC.Read the Free Report