Coding

The application built after the completion of this tutorial provides APIs that allow users to manage a digital library. Users can list all books stored, search any specific book using the filters provided, add new books, update the book's genre, and delete books. They can also list, add, update, or delete book genres. Design and Definition We have the following tables already created in the database. SQL CREATE TABLE genre ( ID INT NOT NULL AUTO_INCREMENT, NAME VARCHAR(100) NOT NULL, DESCRIPTION VARCHAR(300), PRIMARY KEY (ID) ); SQL CREATE TABLE book ( ID INT NOT NULL AUTO_INCREMENT, NAME VARCHAR(100) NOT NULL, AUTHOR_NAME VARCHAR(100) NOT NULL, AUTHOR_SURNAME VARCHAR(100) NOT NULL, EDITORIAL VARCHAR(100), PRIMARY KEY (ID) ); SQL CREATE TABLE book_genres ( BOOK_ID INT NOT NULL, GENRE_ID INT NOT NULL, PRIMARY KEY (BOOK_ID, GENRE_ID) ); In order to provide all functionalities described, we are going to develop 2 APIs in this application, one for books and other for genres management. Book API GET /bookGET /book/id/{id}GET /book?name=XX&author_name=XX&author_surname=XX&editorial=XX&genres=XXPOST /bookDELETE /bookPATCH /bookGenre API GET /genreGET /genre/id/{id}GET /genre?name=XXPOST /genrePATCH /genreDELETE /genre Tutorial Chapters Now that we have defined what and how we are going to do it, we can divide the work that needs to be done into different tasks. Each task needs the previous one to be completed because in all of them we are adding bit by bit the complexity needed to provide all requirements asked. Part 1: Project setup and database configuration. Create the application skeleton, define entities and database configuration.Part 2: Controller creation and API definitions. Define endpoints and add swagger configuration and annotations.Part 3: CRUD Service. Develop services for the CRUD operations, map entities to DTO's and inject the services to the controller.Part 4: Filter and pagination.Part 5: Spring Cache. Configure cacheable methods with auto-refresh and on-demand cache refresh.Part 6: Error management and error messages internationalization. For this tutorial, we are using Spring Boot versión 3.4.0, Java 21 and Maven to build a simple Spring Boot CRUD service using H2 in-memory database. We can build our spring archetype using the following page: https://start.spring.io/ Project Setup and Database Configuration Dependencies In our pom.xml, we need to add the following dependencies: XML <dependency> <groupId>com.h2database</groupId> <artifactId>h2</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-jpa</artifactId> </dependency> Here is a full pom.xml functional file with the basic dependencies needed for this tutorial. XML <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <parent> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-parent</artifactId> <version>3.4.0</version> <relativePath/> <!-- lookup parent from repository --> </parent> <groupId>es.lili.simple.samples</groupId> <artifactId>archetype</artifactId> <version>0.0.1-SNAPSHOT</version> <name>archetype</name> <description>Archetype Spring Boot</description> <properties> <java.version>21</java.version> <mapstruct.version>1.6.0.Beta1</mapstruct.version> <springdoc-webmvc.version>2.7.0</springdoc-webmvc.version> <springdoc-webflux.version>2.7.0</springdoc-webflux.version> <lombok-mapstruct-binding.version>0.2.0</lombok-mapstruct-binding.version> <spring-cloud-starter-config.version>4.2.0</spring-cloud-starter-config.version> </properties> <dependencies> <!--STARTERS--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webflux</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-validation</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-config</artifactId> <version>${spring-cloud-starter-config.version}</version> </dependency> <!--CONFIGURATION--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-configuration-processor</artifactId> <optional>true</optional> </dependency> <!--DATABASE--> <dependency> <groupId>com.h2database</groupId> <artifactId>h2</artifactId> <scope>runtime</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-jpa</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-cache</artifactId> </dependency> <!--DEVELOPER TOOLS--> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency> <dependency> <groupId>org.mapstruct</groupId> <artifactId>mapstruct</artifactId> <version>${mapstruct.version}</version> </dependency> <!--TEST--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>io.projectreactor</groupId> <artifactId>reactor-test</artifactId> <scope>test</scope> </dependency> <!--API DOCUMENTATION--> <dependency> <groupId>org.springdoc</groupId> <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId> <version>${springdoc-webmvc.version}</version> </dependency> <dependency> <groupId>org.springdoc</groupId> <artifactId>springdoc-openapi-starter-webflux-ui</artifactId> <version>${springdoc-webflux.version}</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>${java.version}</source> <target>${java.version}</target> <annotationProcessorPaths> <path> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>${lombok.version}</version> </path> <path> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-configuration-processor</artifactId> </path> <path> <groupId>org.mapstruct</groupId> <artifactId>mapstruct-processor</artifactId> <version>${mapstruct.version}</version> </path> <path> <groupId>org.projectlombok</groupId> <artifactId>lombok-mapstruct-binding</artifactId> <version>${lombok-mapstruct-binding.version}</version> </path> </annotationProcessorPaths> </configuration> </plugin> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <excludes> <exclude> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> </exclude> </excludes> </configuration> </plugin> </plugins> </build> </project> Project Structure Define the application packages and create empty classes in order to have a skeleton to develop the functionalities defined. Entities These are the entities defined for this application. BookEntity Java @Setter @Getter @Builder @Entity @NoArgsConstructor @AllArgsConstructor @Table(name = "book") public class BookEntity { @Id @Column(name = "id") private Long id; @Column(name = "name") private String name; @Column(name = "authorName") private String authorName; @Column(name = "authorSurname") private String authorSurname; @Column(name = "editorial") private String editorial; } GenreEntity Java @Setter @Getter @Builder @Entity @NoArgsConstructor @AllArgsConstructor @Table(name = "genre") public class GenreEntity { @Id @GeneratedValue(strategy=GenerationType.AUTO) private Long id; private String name; private String description; } BookGenreEntity Java @Setter @Getter @Builder @Entity @NoArgsConstructor @AllArgsConstructor @Table(name = "book_genres") @IdClass(BookGenreEntityPk.class) public class BookGenreEntity { @Id @Column(name = "book_id") private Long bookId; @Id @Column(name = "genre_id") private Long genreId; } @Setter @Getter @Builder @NoArgsConstructor @AllArgsConstructor public class BookGenreEntityPk { private Long bookId; private Long genreId; } Configuration We need to modify our application.yml/application.properties so that Spring JPA autoconfigures and create all the needed beans for the H2 database while starting up. In my case, I have both files. The application.yml file aims to the application-{PROFILE}.properties depending on the spring.profiles.active property. You can skip this and only have one of the properties files with the values. The resources folder has the following structure: application.yml YAML spring: config: import: "optional:configserver:http://localhost:8888/" profiles: active: ${ACTIVE_PROFILE:local} application: name: lili-h2-sample datasource: url: ${env.ds-config.url} username: ${env.ds-config.username} password: ${env.ds-config.password} driverClassName: ${env.ds-config.driver} jpa: database-platform: ${env.jpa-config.database-platform} defer-datasource-initialization: ${env.jpa-config.defer-datasource-initialization} #If we add this sql.init properties we MUST have a data.sql file #containing SQL sentences to populate the database sql: init: data-locations: ${env.sql-config.init.data-location} mode: ${env.sql-config.init.mode} platform: ${env.sql-config.init.platform} h2: console: enabled: ${env.h2-config.console.enabled} path: ${env.h2-config.console.path} settings.trace: ${env.h2-config.console.settings.trace} settings.web-allow-others: ${env.h2-config.console.settings.web-allow-others} application-local.properties Properties files env.ds-config.url=jdbc:h2:mem:mydb env.ds-config.username=sa env.ds-config.password=sa env.ds-config.driver=org.h2.Driver #Defines the file location that will be used to populate the database env.sql-config.init.data-location=classpath:h2data/data.sql #Must be true to populate the database and the file specified #in the property above MUST exist and have valid SQL INSERT sentences. env.sql-config.init.mode=always env.sql-config.init.platform=h2 env.jpa-config.database-platform=org.hibernate.dialect.H2Dialect env.jpa-config.defer-datasource-initialization=true env.h2-config.console.enabled=true env.h2-config.console.path=/h2-console env.h2-config.console.settings.trace=false env.h2-config.console.settings.web-allow-others=false When we configure the url, using the :mem: we are indicating we are using in-memory database, this means that all changes in database data during the execution will be lost when the application is stopped. We can save the changes made during execution using file-embedded configuration. The URL then should look like this. The file configured in the URL, in this example C:/data/demodb, is where all the database data will be stored. Properties files spring.datasource.url=jdbc:h2:file:C:/data/demodb Database Population There are different ways to create the tables and schemas needed. Auto. Classess annotated with @Entity are read during start-up to create the tables in the database.File configuration. We need a SQL file that contains all creation sentences needed and indicates its location in the properties file.Console. Once our application is up, we open h2 console and manually create the database schema. Table Initialization — Auto In this example, following the properties file provided above, we let Spring automatically create the tables needed in this application. After our app starts, we can browse our database, accessing the URL configured in the property: Properties files spring.h2.console.path: /h2-console When accessing http://localhost:8080/h2-console this page should appear, where we need to fill in with the same configuration defined in the application properties. After login, we can see all tables that were created during the start-up and are all the classess we annotated with @Entity. Following the same configuration provided, the data.sql file MUST exist and have valid SQL sentences to populate the database. Table Initialization: File Configuration It's not recommended to have this auto generation at the same time as the file configuration for the creation of the database schema. People have reported lots of errors during creation or tables badly created. This automatic table generation was possible thanks to the default value of the following property: Properties files spring.jpa.hibernate.ddl-auto=create-drop The create-drop value is set by default and responsible to tell hibernate that when application is started the database should be created and when the application is stopped it should be deleted. We disable the auto generation, setting the value to none. Properties files spring.jpa.hibernate.ddl-auto=none By default, Spring looks for a schema.sql file to generate the database schema, so after the auto is disabled, we add this file in our resources directory. We also need to change the data.sql file to match the database population sql sentences with the new schema provided Sadly, H2 does not read this schema.sql through Spring Boot resources, so we have to change the jdbc URL to add a script run command. The full .properties file is as follows after these changes to load the configuration using schema.sql Properties files spring.jpa.hibernate.ddl-auto=none env.ds-config.url=jdbc:h2:mem:mydb;INIT=RUNSCRIPT FROM 'src/main/resources/h2data/schema.sql'; env.ds-config.username=sa env.ds-config.password=sa env.ds-config.driver=org.h2.Driver env.sql-config.init.data-location=classpath:h2data/data.sql env.sql-config.init.mode=always env.sql-config.init.platform=h2 env.jpa-config.database-platform=org.hibernate.dialect.H2Dialect env.jpa-config.defer-datasource-initialization=false env.h2-config.console.enabled=true env.h2-config.console.path=/h2-console env.h2-config.console.settings.trace=false env.h2-config.console.settings.web-allow-others=false Accesing now to h2 console we see the @Entity classess were ignored, and the table defined in the SQL file is created and populated. Here, we end the first part of this tutorial. At this point, we only have our application with the minimal configuration required to start and our entities defined. The full project is located at following Github Repository. Each part of the tutorial is located on its own branch, with the final, completed application available on the 'develop' branch.

In this tutorial, we consolidated some practical approaches regarding OpenTelemetry and how to use it with Spring Boot. This tutorial is composed of four primary sections: OpenTelemetry practical conceptsSetting up an observability stack with OpenTelemetry Collector, Grafana, Loki, Tempo, and PodmanInstrumenting Spring Boot applications for OpenTelemetryTesting and E2E sample By the end of the tutorial, you should be able to implement the following architecture: OpenTelemetry Practical Concepts As the official documentation states, OpenTelemetry is: An observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logsVendor and tool-agnostic, meaning that it can be used with a broad variety of Observability backends.Focused on the generation, collection, management, and export of telemetry. A major goal of OpenTelemetry is that you can easily instrument your applications or systems, no matter their language, infrastructure, or runtime environment. Monitoring, Observability, and METL To keep things short, monitoring is the process of collecting, processing, and analyzing data to track the state of a (information) system. Then, monitoring is going to the next level, to actually understand the information that is being collected and do something with it, like defining alerts for a given system. To achieve both goals, it is necessary to collect three dimensions of data, specifically: Logs: Registries about processes and applications, with useful data like timestamps and contextMetrics: Numerical data about the performance of applications and application modulesTraces: Data that allow to estabilish the complete route that a given operation traverses through a series of dependent applications Hence, when the state of a given system is altered in some way, we have an Event, which correlates and ideally generates data on the three dimensions. Why Is OpenTelemetry Important, and What Problem Does It Solve? Developers recognize by experience that monitoring and observability are important, either to evaluate the actual state of a system or to do post-mortem analysis after disasters. Hence, it is natural to think that observability has been implemented in various ways. For example, if we think of a system constructed with Java, we have at least the following collection points: Logs: Systemd, /var/log, /opt/tomcat, FluentDMetrics: Java metrics via JMX, OS Metrics, vendor specific metrics via Spring ActuatorTracing: Data via Jaeger or Zipkin tooling in our Java workloads This variety in turn imposes a great amount of complexity in instrumenting our systems to provide information, that a- comes in different formats, from b- technology that is difficult to implement, often with c- solutions that are too tied to a given provider or in the worst cases, d- technologies that only work with certain languages/frameworks. And that's the magic about the OpenTelemetry proposal: by creating a working group under the CNCF umbrella the project can provide useful things like: Common protocols that vendors and communities can implement to talk to each otherStandards for software communities to implement instrumentation in libraries and frameworks to provide data in OpenTelemetry formatA collector able to retrieve/receive data from diverse origins compatible with OpenTelemetry, process it and send it to...Analysis platforms, databases, and cloud vendors able to receive the data and provide added value over it In short, OpenTelemetry is the reunion of various great monitoring ideas that overlapping software communities can implement to facilitate the burden of monitoring implementations. OpenTelemetry Data Pipeline For me, the easiest way to think about OpenTelemetry concepts is a data pipeline, in this data pipeline you need to Instrument your workloads to push (or offer) the telemetry data to a processing/collecting element — i.e., OpenTelemetry Collector-Configure OpenTelemetry Collector to receive or pull the data from diverse workloadsConfigure OpenTelemetry Collector to process the data — i.e., adding special tags, filtering dataConfigure OpenTelemetry Collector to push (or offer) the data to compatible backendsConfigure and use the backends to receive (or pull) the data from the collector to allow analysis, alarms, AI... pretty much any case that you can think about with data Setting up an observability stack with OpenTelemetry Collector, Grafana, Prometheus, Loki, Tempo and Podman As OpenTelemetry got popular various vendors have implemented support for it, to mention a few: Self-hosted platforms ElasticGrafanaHyperDX Cloud platforms AmazonOracle CloudSplunkDatadog Hence, for development purposes, it is always useful to know how to bootstrap a quick observability stack able to receive and show OpenTelemetry capabilities. For this purpose, we will use the following elements: Prometheus as a time-series database for metricsLoki as a logs platformTempo as a tracing platformGrafana as a web UI And of course OpenTelemetry collector. This example is based on various Grafana examples, with a little bit of tweaking to demonstrate the different ways of collecting, processing and sending data to backends. OpenTelemetry Collector As stated previously, OpenTelemetry collector acts as an intermediary that receives/pull information from data sources, processes this information and, forwards the information to destinations like analysis platforms or even other collectors. The collector is able to do this either with compliant workloads or via plugins that talk with the workloads using proprietary formats. As the plugins collection can be increased or decreased, vendors have created their own distributions of OpenTelemetry collectors, for reference I've used successfully in the real world: Amazon ADOTSplunk Distribution of OpenTelemetry CollectorGrafana AlloyOpenTelemetry Collector (the reference implementation) You can find a complete list directly on the OpenTelemetry website. For this demonstration, we will create a data pipeline using the contrib version of the reference implementation, which provides a good amount of receivers, exporters, and processors. In our case, Otel configuration is designed to: Receive data from Spring Boot workloads (ports 4317 and 4318)Process the data, adding a new tag to metricsExpose an endpoint for Prometheus scraping (port 8889)Send logs to Loki (port 3100) using otlphttp formatSend traces to Tempo (port 9411) using otlp formatExposes a rudimentary dashboard from the collector, called zpages. Very useful for debugging. otel-config.yaml YAML receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: attributes: actions: - key: team action: insert value: vorozco exporters: debug: prometheus: endpoint: "0.0.0.0:8889" otlphttp: endpoint: http://loki:3100/otlp otlp: endpoint: tempo:4317 tls: insecure: true service: extensions: [zpages] pipelines: metrics: receivers: [otlp] processors: [attributes] exporters: [debug,prometheus] traces: receivers: [otlp] exporters: [debug, otlp] logs: receivers: [otlp] exporters: [debug, otlphttp] extensions: zpages: endpoint: "0.0.0.0:55679" Prometheus Prometheus is a well known analysis platform, that among other things offers dimensional data and a performant time-series storage. By default, it works as a metrics scrapper, then, workloads provide a http endpoint offering data using the Prometheus format. For our example, we configured Otel to offer metrics to the prometheus host via port 8889. YAML prometheus: endpoint: "prometheus:8889" Then, whe need to configure Prometheus to scrape the metrics from the Otel host. You would notice two ports, the one that we defined for the active workload data (8889) and another for metrics data for the collector itself (8888). prometheus.yml YAML scrape_configs: - job_name: "otel" scrape_interval: 10s static_configs: - targets: ["otel:8889"] - targets: ["otel:8888"] It is worth highlighting that Prometheus also offers a way to ingest information instead of scrapping it, and, the official support for OpenTelemetry ingestion is coming on the new versions. Loki As described in the website, Loki is a specific solution for log aggregation heavily inspired by Prometheus, with the particular design decision to NOT format in any way the log contents, leaving that responsibility to the query system. To configure the project for local environments, the project offers a configuration that is usable for most of the development purposes. The following configuration is an adaptation to preserve the bare minimum to work with temporal files and memory. loki.yaml YAML auth_enabled: false server: http_listen_port: 3100 grpc_listen_port: 9096 common: instance_addr: 127.0.0.1 path_prefix: /tmp/loki storage: filesystem: chunks_directory: /tmp/loki/chunks rules_directory: /tmp/loki/rules replication_factor: 1 ring: kvstore: store: inmemory query_range: results_cache: cache: embedded_cache: enabled: true max_size_mb: 100 schema_config: configs: - from: 2020-10-24 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h ruler: alertmanager_url: http://localhost:9093 limits_config: allow_structured_metadata: true Then, we configure an exporter to deliver the data to the loki host using oltphttp format. YAML otlphttp: endpoint: http://loki:3100/otlp Tempo In similar fashion than Loki, Tempo is an Open Source project created by grafana that aims to provide a distributed tracing backend. On a personal note, for me besides performance it shines for being compatible not only with OpenTelemetry, it can also ingest data in Zipkin and Jaeger formats. To configure the project for local environments, the project offers a configuration that is usable for most of the development purposes. The following configuration is an adaptation to remove the metrics generation and simplify the configuration, however with this we loose the service graph feature. tempo.yaml YAML stream_over_http_enabled: true server: http_listen_port: 3200 log_level: info query_frontend: search: duration_slo: 5s throughput_bytes_slo: 1.073741824e+09 metadata_slo: duration_slo: 5s throughput_bytes_slo: 1.073741824e+09 trace_by_id: duration_slo: 5s distributor: receivers: otlp: protocols: http: grpc: ingester: max_block_duration: 5m # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally compactor: compaction: block_retention: 1h # overall Tempo trace retention. set for demo purposes storage: trace: backend: local # backend configuration to use wal: path: /var/tempo/wal # where to store the wal locally local: path: /var/tempo/blocks Then, we configure an exporter to deliver the data to Tempo host using oltp/grpc format. YAML otlp: endpoint: tempo:4317 tls: insecure: true Grafana Loki, Tempo and (to some extent) Prometheus are data storages, but we still need to show this data to the user. Here, Grafana enters the scene. Grafana offers a good selection of analysis tools, plugins, dashboards, alarms, connectors and a great community that empowers observability. Besides having a great compatibility with Prometheus, it offers of course a perfect compatibility with their other offerings. To configure Grafana, you just need to plug compatible datasources, and the rest of the work will be on the web ui. grafana.yaml YAML apiVersion: 1 datasources: - name: Otel-Grafana-Example type: prometheus url: http://prometheus:9090 editable: true - name: Loki type: loki access: proxy orgId: 1 url: http://loki:3100 basicAuth: false isDefault: true version: 1 editable: false - name: Tempo type: tempo access: proxy orgId: 1 url: http://tempo:3200 basicAuth: false version: 1 editable: false apiVersion: 1 uid: tempo Podman (or Docker) At this point, you may have noticed that I've referred to the backends using single names. This is because I intend to set these names using a Podman Compose deployment. otel-compose.yml YAML version: '3' services: otel: container_name: otel image: otel/opentelemetry-collector-contrib:latest command: [--config=/etc/otel-config.yml] volumes: - ./otel-config.yml:/etc/otel-config.yml ports: - "4318:4318" - "4317:4317" - "55679:55679" prometheus: container_name: prometheus image: prom/prometheus command: [--config.file=/etc/prometheus/prometheus.yml] volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml ports: - "9091:9090" grafana: container_name: grafana environment: - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin image: grafana/grafana volumes: - ./grafana.yml:/etc/grafana/provisioning/datasources/default.yml ports: - "3000:3000" loki: container_name: loki image: grafana/loki:3.2.0 command: -config.file=/etc/loki/local-config.yaml volumes: - ./loki.yaml:/etc/loki/local-config.yaml ports: - "3100" tempo: container_name: tempo image: grafana/tempo:latest command: [ "-config.file=/etc/tempo.yaml" ] volumes: - ./tempo.yaml:/etc/tempo.yaml ports: - "4317" # otlp grpc - "4318" At this point, the compose description is pretty self-descriptive, but I would like to highlight some things: Some ports are open to the host -e.g. 4318:4318 - while others are closed to the default network that compose will be created among containers -e.g. 3100-This stack is designed to avoid any permanent data. Again, this is my personal way to boot quickly an observability stack to allow tests during deployment. To make it ready for production, you probably would want to preserve the data in some volumes Once the configuration is ready, you can launch it using the compose file Shell cd podman podman compose -f otel-compose.yml up If the configuration is ok, you should have five containers running without errors. Instrumenting Spring Boot Applications for OpenTelemetry As part of my daily activities, I was in charge of a major implementation of all these concepts. Hence, it was natural for me to create a proof of concept that you can find at my GitHub. For demonstration purposes, we have two services with different HTTP endpoints: springboot-demo:8080- Useful to demonstrate local and database tracing, performance, logs and OpenTelemetry instrumentation /books — A books CRUD using Spring Data/fibo — A Naive Fibonacci implementation that generates CPU load and delays/log — Which generate log messages using the different SLF4J levelsspringboot-client-demo:8081- Useful to demonstrate tracing capabilities, Micrometer instrumentation and Micrometer Tracing instrumentation /trace-demo - A quick OpenFeing client that invokes books GetAll Books demo Instrumentation Options Given the popularity of OpenTelemetry, developers can also expect multiple instrumentation options. First of all, the OpenTelemetry project offers a framework-agnostic instrumentation that uses bytecode manipulation, for this instrumentation to work you need to include a Java Agent via Java Classpath. In my experience this instrumentation is preferred if you don't control the workload or if your platform does not offer OpenTelemetry support at all. However, instrumentation of workloads can become really specific — e.g. instrumentation of a Database pool given a particular IoC mechanism. For this, the Java world provides a good ecosystem, for example: QuarkusHelidonPayara And, of course, Spring Boot. Spring Boot is a special case with TWO major instrumentation options OpenTelemetry's Spring Boot starterMicrometer and Micrometer Tracing Both options use Spring concepts like decorators and interceptors to capture and send information to the destinations. The only rule is to create the clients/services/objects in the Spring way (hence via Spring IoC). I've used both successfully, and my heavily opinionated conclusion is the following: Micrometer collects more information about spring metrics. Besides the OpenTelemetry backend, it supports a plethora of backends directly without any collector intervention. If you cannot afford a collector, this is the way. From a Micrometer perspective, OpenTelemetry is just another backend.Micrometer Tracing is the evolution of Spring Cloud Sleuth, hence, if you have workloads with Spring Boot 2 and 3, you have to support both tools (or maybe migrate everything to Spring boot 3?)The Micrometer family does not offer a way to collect logs and send these to a backend, hence devs have to solve this by using an appender specific to your logging library. On the other hand OpenTelemetry Spring Boot starter offers this out of the box if you use Spring Boot default (SLF4J over Logback) As these libraries are mutually exclusive, if the decision were mine, I would pick OpenTelemetry's Spring Boot starter. It offers logs support OOB and also a bridge for micrometer Metrics. Instrumenting springboot-demo With OpenTelemetry SpringBoot Starter As always, it is also good to consider the official documentation. Otel instrumentation with the Spring started is activated in three steps: You need to include both OpenTelemetry Bom and OpenTelemetry dependency. If you are planning to also use micrometer metrics, it is also a good idea to include Spring Actuator XML <dependencyManagement> <dependencies> <dependency> <groupId>io.opentelemetry.instrumentation</groupId> <artifactId>opentelemetry-instrumentation-bom</artifactId> <version>2.10.0</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> ... <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.opentelemetry.instrumentation</groupId> <artifactId>opentelemetry-spring-boot-starter</artifactId> </dependency> There is a set of optional libraries and adapters that you can configure if your workloads already diverged from the "Spring Way" You need to activate (or not) the dimensions of observability (metrics, traces and logs). Also, you can fine-tune the exporting parameters like ports, URLs, or exporting periods. Either by using Spring Properties or env variables Properties files #Configure exporters otel.logs.exporter=otlp otel.metrics.exporter=otlp otel.traces.exporter=otlp #Configure metrics generation otel.metric.export.interval=5000 #Export metrics each five seconds otel.instrumentation.micrometer.enabled=true #Enabe Micrometer metrics bridge Instrumenting springboot-client-demo With Micrometer and Micrometer Tracing Again, this instrumentation does not support logs exporting. Also, it is a good idea to check the latest documentation for Micrometer and Micrometer Tracing. As in the previous example, you need to enable the Spring Actuator (which includes Micrometer). As OpenTelemetry is just a backend from Micrometer's perspective, so you just need to enable the corresponding OTLP registry, which will export metrics to localhost by default. XML <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-otlp</artifactId> </dependency> In a similar way, once Actuator is enabled, you just need to add support for the tracing backend. XML <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> </dependency> Finally, you can fine-tune the configuration using Spring properties. For example, you can decide if 100% of traces are reported or how often the metrics are reported to the backend. Properties files management.otlp.tracing.endpoint=http://localhost:4318/v1/traces management.otlp.tracing.timeout=10s management.tracing.sampling.probability=1 management.otlp.metrics.export.url=http://localhost:4318/v1/metrics management.otlp.metrics.export.step=5s management.opentelemetry.resource-attributes."service-name"=${spring.application.name} Testing and E2E Sample Generating Workload Data The POC provides the following structure ├── podman # Podman compose config files ├── springboot-client-demo #Spring Boot Client instrumented with Actuator, Micrometer and MicroMeter tracing └── springboot-demo #Spring Boot service instrumented with OpenTelemetry Spring Boot Starter The first step is to boot the observability stack we created previously. Shell cd podman podman compose -f otel-compose.yml up This will provide you with an instance of Grafana on port 3000 Then, it is time to boot the first service!. You only need Java 21 on the active shell: Shell cd springboot-demo mvn spring-boot:run If the workload is properly configured, you will see the following information on the OpenTelemetry container standard output. Which basically says you are successfully reporting data. Shell [otel] | 2024-12-01T22:10:07.730Z info Logs {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 24} [otel] | 2024-12-01T22:10:10.671Z info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 64, "data points": 90} [otel] | 2024-12-01T22:10:10.672Z info Traces {"kind": "exporter", "data_type": "traces", "name": "debug", "resource spans": 1, "spans": 5} [otel] | 2024-12-01T22:10:15.691Z info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 65, "data points": 93} [otel] | 2024-12-01T22:10:15.833Z info Metrics {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 65, "data points": 93} [otel] | 2024-12-01T22:10:15.835Z info Logs {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 5} The data is being reported over the OpenTelemetry ports (4317 and 4318), which are open from Podman to the host. By default, all telemetry libraries report to localhost, but this can be configured for other cases like FaaS or Kubernetes. Also, you can verify the reporting status in ZPages. Finally, let's do the same withthe Spring Boot client: Shell cd springboot-client-demo mvn spring-boot:run As described in the previous section, I created a set of interactions to: Generate CPU workload using Naive fibonacci: Shell curl http://localhost:8080/fibo\?n\=45 Generate logs in different levels: Shell curl http://localhost:8080/fibo\?n\=45 Persist data using a CRUD: Shell curl -X POST --location "http://localhost:8080/books" \ -H "Content-Type: application/json" \ -d '{ "author": "Miguel Angel Asturias", "title": "El señor presidente", "isbn": "978-84-376-0494-7", "publisher": "Editorial planeta" }' And then retrieve the data using a secondary service: Shell curl http://localhost:8081/trace-demo This asciicast shows the interaction: https://asciinema.org/a/692968 Grafana Results Once the data is accessible by Grafana, the what to do with data is up to you. Again, you could: Create dashboardsConfigure alarmsConfigure notifications from alarms The quickest way to verify if the data is reported correctly is to verify directly in Grafana explore. First, we can check some metrics like system_cpu_usage and filter by service name. In this case I used springboot-demo which has the CPU demo using naive fibonacci, I can even filter by my own tag (which was added by Otel processor): In the same way, logs are already stored in Loki: Finally, we could check the whole trace, including both services and interaction with H2 RDBMS: Conclusion In conclusion, implementing OpenTelemetry with Spring Boot provides a robust solution for observability, offering comprehensive insights through metrics, traces, and logs. By integrating tools like Prometheus, Grafana, Tempo, and Loki, you can easily monitor and troubleshoot your applications. Whether you choose OpenTelemetry's Spring Boot starter or Micrometer, both provide powerful instrumentation options, allowing you to gain deeper visibility and improve application performance.

In today's microservices-driven world, managing traffic smartly is just as crucial as deploying the services themselves. As your system grows, so do the risks — like overuse, misuse, and cascading failures. And if you're running multi-tenant services, it's essential to enforce request limits for each customer. That’s where rate limiting in a service mesh like Istio can make a big difference. In this post, we’ll explore why rate limiting is important in Istio and show you how to set it up effectively. Why Rate Limiting Matters in Istio Why Was It Important for Us? This is in continuation of the incident that we faced, which is detailed in How I Made My Liberty Microservices Load-Resilient. One of the findings during the incident was the missing rate limiting in the Istio ingress private gateway. Here are the challenges that we faced: Challenge 1: Backend Overload During High-Traffic Events: Our services — especially user-facing APIs — would get hit hard during high traffic. Without any rate controls in place, back-end services and databases would spike, leading to degraded performance or even downtime.Challenge 2: Need for Tenant-Based Throttling: We support multiple customers (tenants). We needed a way to enforce quotas, ensuring one tenant’s overuse wouldn’t impact others. Architecture Here is a simplified version of the application architecture, highlighting the key components involved in the Istio rate limiting. Why Rate Limiting Matters in Istio Rate limiting is a mechanism to control how many requests a service or user can make over a given time interval. In the context of an Istio service mesh, it serves several important purposes: Protecting Backend Services: Without rate limiting, a sudden spike in requests — whether from legitimate users or abusive clients — can overload back-end services and databases. Rate limiting helps prevent service degradation or outages.Fair Usage and Quotas: Ensures fair access in a multi-tenant system among our tenants by enforcing per-user limits.Cost Control: In cloud-native environments, uncontrolled requests can lead to excessive resource usage and unexpected costs. Rate limiting helps contain those costs.Security and Abuse Prevention: Blocking excessive requests from abusive users can mitigate denial-of-service (DoS) attacks or brute-force attempts.Improved Reliability: By limiting how fast clients can make requests, services stay more responsive and predictable, even under heavy load. With implementing rate limiting in Istio, there are two places where it is applied. One for the public endpoint customers is done using Cloudflare, and another for private endpoint customers is using Istio. Local Rate Limiting vs Global Rate Limiting When implementing rate limiting in an Istio mesh, it’s important to understand the difference between local and global rate limiting, as they serve different use cases and come with distinct trade-offs. Local rate limiting is enforced individually on each Envoy sidecar proxy. The rate limit is applied at the instance level, meaning each pod or replica gets its own independent limit. Global rate limiting is centrally managed using an external RLS. Envoy sidecars query this central service to decide whether to allow or deny a request, ensuring cluster-wide enforcement. Note: Local rate limiting in Kubernetes is enforced per pod. This means the token or request count value you configure applies to each individual pod independently. For example, if you have 5 pods and want to enforce a global limit of 100 requests per 60 seconds, you should configure the local rate limit token value to 20 per pod (i.e., 100 / 5 = 20). This calculation assumes that the load balancer distributes traffic evenly across all pods. Feature Local Rate Limiting Global Rate Limiting Enforced At Individual Envoy proxy Central Rate Limit Service Scope Per pod/instance Across all instances Infrastructure Required None Redis + Rate Limit Service Granularity Basic (fixed per pod) Fine-grained (per-user, route) Use Case Simple protection Multi-tenant, API Gateway Scalability Awareness No Yes Let's explore how to configure Local Rate Limiting in a Kubernetes environment. Steps to Set Up Local Rate Limiting in Istio Step 1: Create the EnvoyFilter for Local Rate Limiting Descriptors are keys used to match requests and apply limits. Example config.yamlfor Envoy: YAML apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: ingress-gateway-ratelimit-envoyfilter namespace: <namespace> spec: configPatches: - applyTo: HTTP_FILTER match: context: GATEWAY listener: filterChain: filter: name: envoy.filters.network.http_connection_manager patch: operation: INSERT_FIRST value: name: envoy.filters.http.local_ratelimit typed_config: '@type': type.googleapis.com/udpa.type.v1.TypedStruct type_url: >- type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit value: stat_prefix: http_local_rate_limiter - applyTo: VIRTUAL_HOST match: context: GATEWAY routeConfiguration: gateway: <namespace>/<gateway-name> portName: https-custom-regional portNumber: 443 patch: operation: MERGE value: rate_limits: - actions: - request_headers: descriptor_key: method header_name: ":method" typed_per_filter_config: envoy.filters.http.local_ratelimit: '@type': type.googleapis.com/udpa.type.v1.TypedStruct type_url: >- type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit value: always_consume_default_token_bucket: false descriptors: - entries: - key: method value: "GET" token_bucket: fill_interval: 60s max_tokens: 10 tokens_per_fill: 10 - entries: - key: method value: "POST" token_bucket: fill_interval: 60s max_tokens: 2 tokens_per_fill: 2 - entries: - key: method value: "PATCH" token_bucket: fill_interval: 60s max_tokens: 4 tokens_per_fill: 4 filter_enabled: default_value: denominator: HUNDRED numerator: 100 runtime_key: local_rate_limit_enabled filter_enforced: default_value: denominator: HUNDRED numerator: 100 runtime_key: local_rate_limit_enforced response_headers_to_add: - append: false header: key: x-local-rate-limit value: 'true' - append: false header: key: retry-after value: '60' stat_prefix: http_local_rate_limiter token_bucket: fill_interval: 60s max_tokens: 40 tokens_per_fill: 40 workloadSelector: labels: <gateway-label>: <gateway-label> Provide the Gateway name and namespace in route_configuration section, as we are applying the rate limiting for Gateway pods. WorkLoadSelector will have the pods labels where you want to apply rate limiting. The token bucket algorithm is used to define a bucket that holds a limited number of "tokens" representing allowed requests. The bucket is continuously refilled with tokens at a specific rate (tokens per second or minute). When a request arrives, it consumes a token from the bucket. If the bucket is empty, subsequent requests are rejected (rate-limited) until the bucket refills. Imagine a bucket with a capacity of 10 tokens, refilled with 1 token every second. If 10 requests arrive within 1 second, they will all be allowed, but the 11th request will be rejected until the bucket refills. Istio supports both local and global rate limiting. Local rate limiting applies to individual service instances, while global rate limiting applies across all instances of a service. In local rate limiting, each Envoy proxy (representing a service instance) has its own token bucket, and the token bucket quota is not shared among the replicas. In global rate limiting, the token bucket quota is shared among all Envoy proxies, meaning that the service will only accept a certain number of requests in total, regardless of the number of replicas. In the above configuration, token_bucket is defined in the (HTTP_ROUTE) patch, which includes a typed_per_filter_config for the envoy.filters.http.local_ratelimit local envoy filter for routes to virtual host inbound for https at port 443. max_tokens specifies the maximum number of tokens in the bucket, representing the maximum allowed requests within a certain time frame. tokens_per_fill indicates the number of tokens added to the bucket with each fill, essentially the rate at which tokens are replenished. fill_interval defines the time interval at which the bucket is replenished with tokens. In the example above, we configured rate limits for specific HTTP methods like GET, POST, and PATCH. You can add or remove methods in this section based on your requirements. The token values assigned under each method define the rate limit for requests using that method. If a request uses a method that isn't explicitly listed, the default token value (configured below stat_prefix) will apply instead. Step 1.1: Define Rate Limit per Instance (Optional) If you need tenant-based rate limiting, enable it based on the instance ID (or any other value as applicable) of the tenant instead of the above configuration mentioned in Step 1. Below is the updated configuration to rate limit based on the instance ID. The instance ID, in our case, is part of the request URL. So, the below configuration has two noticeable updates: Rate-limiting descriptors are added with entries to the instance ID value as a key-value pair. These are the instance IDs, which generally have a heavy load, and these are identified based on the history of requests for rate limits.Source code addition: function envoy_on_request is added to retrieve the instance ID from the request URL and to match the instance IDs identified in the above one. YAML apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: ingress-gateway-ratelimit-envoyfilter namespace: <namespace> spec: configPatches: - applyTo: HTTP_FILTER match: context: GATEWAY listener: filterChain: filter: name: envoy.filters.network.http_connection_manager patch: operation: INSERT_BEFORE value: name: envoy.filters.http.local_ratelimit typed_config: '@type': type.googleapis.com/udpa.type.v1.TypedStruct type_url: >- type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit value: stat_prefix: http_local_rate_limiter - applyTo: VIRTUAL_HOST match: context: GATEWAY routeConfiguration: gateway: <namespace>/<gateway-name> portName: https-custom-regional portNumber: 443 patch: operation: MERGE value: rate_limits: - actions: - request_headers: descriptor_key: rate-limit header_name: x-instance-id typed_per_filter_config: envoy.filters.http.local_ratelimit: '@type': type.googleapis.com/udpa.type.v1.TypedStruct type_url: >- type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit value: always_consume_default_token_bucket: false descriptors: - entries: - key: rate-limit value: ```"<instance-id>"``` token_bucket: fill_interval: 60s max_tokens: 2 tokens_per_fill: 2 - entries: - key: rate-limit value: ```"<instance-id-2>"``` token_bucket: fill_interval: 60s max_tokens: 5 tokens_per_fill: 5 filter_enabled: default_value: denominator: HUNDRED numerator: 100 runtime_key: local_rate_limit_enabled filter_enforced: default_value: denominator: HUNDRED numerator: 100 runtime_key: local_rate_limit_enforced response_headers_to_add: - append: false header: key: x-local-rate-limit value: 'true' - append: false header: key: retry-after value: '60' stat_prefix: http_local_rate_limiter token_bucket: fill_interval: 60s max_tokens: 40 tokens_per_fill: 40 - applyTo: HTTP_FILTER match: context: GATEWAY listener: filterChain: filter: name: envoy.filters.network.http_connection_manager patch: operation: INSERT_BEFORE value: name: envoy.filters.http.lua typed_config: '@type': type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua default_source_code: inline_string: | function envoy_on_request(request_handle) local path = request_handle:headers():get(":path") or "nil" request_handle:logInfo("Request received: " .. path) local instance_id = path:match("/instances/([^/]+)/") local rate_limited_instance = ```"<instance-id>"``` local rate_limited_instance_2 = ```"<instance-id-2>"``` local method = request_handle:headers():get(":method") if instance_id then request_handle:logInfo("Extracted Instance ID: " .. instance_id) else request_handle:logInfo("No Instance ID found in path") end if instance_id == rate_limited_instance or instance_id == rate_limited_instance_2 then request_handle:logInfo("method: "..request_handle:headers():get(":method")) if method == "GET" then request_handle:logInfo("Applying rate limiting for instance_id: " .. instance_id) request_handle:headers():add("x-instance-id", tostring(instance_id)) request_handle:logInfo("instance_header: "..request_handle:headers():get("x-instance-id")) else request_handle:logInfo("Not get call for instance_id: " .. tostring(instance_id)) end else request_handle:logInfo("No rate limiting for instance_id: " .. tostring(instance_id)) end end workloadSelector: labels: <gaterway-label>: <gaterway-label> Note: Since we used a custom Envoyfilter, we need not deploy Redis, which is otherwise used for metadata like counters and TTLs. Step 2: Test the Configuration Send requests to your service and observe if the rate limits are triggered. You can check the logs from the pods where you have applied rate limiting. Step 3: Monitor the Requests That Are Rate Limited Today, we monitor the rate limits from logs. But in the future, we would enable Prometheus/Grafana dashboards to observe the rate-limited traffic. Conclusion Rate limiting in Istio is not just a performance enhancement — it’s a strategic traffic control mechanism that protects your services, ensures fairness, and supports system scalability. Whether you're securing an API gateway or managing internal service communication, rate limiting should be part of your Istio playbook.

By Josephine Eskaline Joyce CORE

In the world of data analysis, "slow queries" are like workplace headaches that just won't go away. Recently, I've met quite a few data analysts who complain about queries running for hours without results, leaving them staring helplessly at the spinning progress bar. Last week, I ran into an old friend who was struggling with the performance of a large table JOIN. "The query speed is slower than a snail, and my boss is driving me crazy..." he said with a frustrated look. As a seasoned database optimization expert with years of experience on the front lines, I couldn't help but smile: "JOIN performance is slow because you don't understand its nature. Just like in martial arts, understanding how to use force effectively can make all the difference." Today, let's dive into the world of JOIN operations in Doris and see how you can transform your queries into "lightning-fast" operations that will impress your boss! Doris JOIN Secret: Performance Optimization Starts With Choosing the Right JOIN Strategy Data Analyst Xiao Zhang recently encountered a tricky problem. He was working on a large-scale data analysis task that required joining several massive tables. Initially, he used the most conventional JOIN method, but the query speed was painfully slow, taking hours to complete. This left him in a tough spot — his boss needed the report urgently, and the pressure was mounting. Xiao Zhang turned to his old friend and Doris expert, Old Li, for help. Old Li smiled and said, "The key to JOIN performance is choosing the right JOIN strategy. Doris supports multiple JOIN implementations, and I'll share some secrets with you." The Essence of JOIN In distributed databases, JOIN operations may seem simple, but they actually involve hidden complexities. They not only need to join tables but also coordinate data flow and computation in a distributed environment. For example, you have two large tables distributed across different nodes, and to perform a JOIN, you need to solve a core problem — how to bring the data to be joined together. This involves data redistribution strategies. Doris' JOIN Arsenal Doris employs two main physical implementations for JOIN operations — Hash Join and Nest Loop Join. Hash Join is like a martial artist's swift sword, completing joins with lightning speed; Nest Loop Join is like a basic skill for a swordsman, simple but widely applicable. Hash Join builds a hash table in memory and can quickly complete equi-JOIN operations. It's like finding teammates in a game of League of Legends — each player is assigned a unique ID, and when needed, the system locates them directly, naturally achieving high efficiency. Nest Loop Join, on the other hand, uses the most straightforward method — iteration. It's like visiting relatives during the Spring Festival, going door-to-door, which is slower but ensures that no one is missed. This method is applicable to all JOIN scenarios, including non-equi-JOINs. The Four Data Distribution Strategies in the JOIN World As a distributed MPP database, Apache Doris needs to shuffle data during the Hash Join process to ensure the correctness of the JOIN results. Old Li pulled out a diagram and showed Xiao Zhang the four data distribution strategies for JOIN operations in Doris: Broadcast Join: The Dominant Player Broadcast Join is like a domineering CEO who replicates the right table's data to every compute node. It's simple and brutal, widely applicable. When the right table's data volume is small, this method is efficient. Network overhead grows linearly with the number of nodes. As shown in the figure, Broadcast Join involves sending all the data from the right table to all nodes participating in the JOIN computation, including the nodes scanning the left table data, while the left table data remains stationary. Each node receives a complete copy of the right table's data (total data volume T(R)), ensuring that all nodes have the necessary data to perform the JOIN operation. This method is suitable for a variety of general scenarios but is not applicable to RIGHT OUTER, RIGHT ANTI, and RIGHT SEMI types of Hash Join. The network overhead is the number of JOIN nodes N multiplied by the right table's data volume T(R). Partition Shuffle: The Conventional Approach Partition Shuffle employs a bidirectional distribution strategy, hashing and distributing both tables based on the JOIN key. The network overhead equals the sum of the data volumes of the two tables. This method is like Tai Chi, emphasizing balance, and is suitable for scenarios where the data volumes of the two tables are similar. This method involves calculating hash values based on the JOIN condition and partitioning the data accordingly. Specifically, the data from both the left and right tables is partitioned based on the hash values calculated from the JOIN condition and then sent to the corresponding partition nodes (as shown in the figure). The network overhead for this method includes the cost of transmitting the left table's data T(S) and the cost of transmitting the right table's data T(R). This method only supports Hash Join operations because it relies on the JOIN condition to perform data partitioning. Bucket Shuffle: The Cost-Effective Approach Bucket Shuffle leverages the bucketing characteristics of the tables, requiring redistribution of only the right table's data. The network overhead is merely the data volume of the right table. It's like a martial artist using the opponent's strength against them, making good use of their advantages. This method is particularly efficient when the left table is already bucketed by the JOIN key. When the JOIN condition includes the bucketing column of the left table, the left table's data remains stationary, and the right table's data is redistributed to the left table's nodes for JOIN, reducing network overhead. When one side of the table participating in the JOIN operation has data that is already hash-distributed according to the JOIN condition column, we can choose to keep the data location of this side unchanged and only move and redistribute the other side of the table. (Here, the "table" is not limited to physically stored tables but can also be the output result of any operator in an SQL query, and we can flexibly choose to keep the data location of the left or right table unchanged and only move the other side.) Taking Doris' physical tables as an example, since the table data is stored through hash distribution calculations, we can directly use this characteristic to optimize the data shuffle process during JOIN operations. Suppose we have two tables that need to be JOINed, and the JOIN column is the bucketing column of the left table. In this case, we do not need to move the left table's data; we only need to redistribute the right table's data according to the left table's bucketing information to complete the JOIN calculation (as shown in the figure). The network overhead for this process mainly comes from the movement of the right table's data, i.e., T(R). Colocate Join: The Ultimate Expert Colocate Join is the ultimate optimization, where data is pre-distributed in the same manner, and no data movement is required during JOIN. It's like a perfectly synchronized partnership, with seamless cooperation. Zero network overhead and optimal performance, but it has the strictest requirements. Similar to Bucket Shuffle Join, if both sides of the table participating in the JOIN are hash-distributed according to the JOIN condition column, we can skip the shuffle process and directly perform the JOIN calculation locally. Here is a simple illustration using physical tables: When Doris creates tables with DISTRIBUTED BY HASH, the data is distributed according to the hash distribution key during data import. If the hash distribution key of the two tables happens to match the JOIN condition column, we can consider that the data of these two tables has been pre-distributed according to the JOIN requirements, i.e., no additional shuffle operation is needed. Therefore, during actual querying, we can directly perform the JOIN calculation on these two tables. Colocate Join Example After introducing the four types of Hash Join in Doris, let's pick Colocate Join, the ultimate expert, for a showdown: In the example below, both tables t1 and t2 have been processed by the GROUP BY operator and output new tables (at this point, both tx and ty are hash-distributed according to c2). The subsequent JOIN condition is tx.c2 = ty.c2, which perfectly meets the conditions for Colocate Join. SQL explain select * from ( -- Table t1 is hash-distributed by c1. After the GROUP BY operator, the data distribution becomes hash-distributed by c2. select c2 as c2, sum(c1) as c1 from t1 group by c2 ) tx join ( -- Table t2 is hash-distributed by c1. After the GROUP BY operator, the data distribution becomes hash-distributed by c2. select c2 as c2, sum(c1) as c1 from t2 group by c2 ) ty on tx.c2 = ty.c2; From the Explain execution plan result below, we can see that the left child node of the 8th Hash Join node is the 7th aggregation node, and the right child node is the 3rd aggregation node, with no Exchange node appearing. This indicates that the data from both the left and right child nodes after aggregation remains in its original location without any data movement and can directly perform the subsequent Hash Join operation locally. SQL +------------------------------------------------------------+ | Explain String(Nereids Planner) | +------------------------------------------------------------+ | PLAN FRAGMENT 0 | | OUTPUT EXPRS: | | c2[#20] | | c1[#21] | | c2[#22] | | c1[#23] | | PARTITION: HASH_PARTITIONED: c2[#10] | | | | HAS_COLO_PLAN_NODE: true | | | | VRESULT SINK | | MYSQL_PROTOCAL | | | | 8:VHASH JOIN(373) | | | join op: INNER JOIN(PARTITIONED)[] | | | equal join conjunct: (c2[#14] = c2[#6]) | | | cardinality=10 | | | vec output tuple id: 9 | | | output tuple id: 9 | | | vIntermediate tuple ids: 8 | | | hash output slot ids: 6 7 14 15 | | | final projections: c2[#16], c1[#17], c2[#18], c1[#19] | | | final project output tuple id: 9 | | | distribute expr lists: c2[#14] | | | distribute expr lists: c2[#6] | | | | | |----3:VAGGREGATE (merge finalize)(367) | | | | output: sum(partial_sum(c1)[#3])[#5] | | | | group by: c2[#2] | | | | sortByGroupKey:false | | | | cardinality=5 | | | | final projections: c2[#4], c1[#5] | | | | final project output tuple id: 3 | | | | distribute expr lists: c2[#2] | | | | | | | 2:VEXCHANGE | | | offset: 0 | | | distribute expr lists: | | | | | 7:VAGGREGATE (merge finalize)(354) | | | output: sum(partial_sum(c1)[#11])[#13] | | | group by: c2[#10] | | | sortByGroupKey:false | | | cardinality=10 | | | final projections: c2[#12], c1[#13] | | | final project output tuple id: 7 | | | distribute expr lists: c2[#10] | | | | | 6:VEXCHANGE | | offset: 0 | | distribute expr lists: | | | | PLAN FRAGMENT 1 | | | | PARTITION: HASH_PARTITIONED: c1[#8] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 06 | | HASH_PARTITIONED: c2[#10] | | | | 5:VAGGREGATE (update serialize)(348) | | | STREAMING | | | output: partial_sum(c1[#8])[#11] | | | group by: c2[#9] | | | sortByGroupKey:false | | | cardinality=10 | | | distribute expr lists: c1[#8] | | | | | 4:VOlapScanNode(345) | | TABLE: tt.t1(t1), PREAGGREGATION: ON | | partitions=1/1 (t1) | | tablets=1/1, tabletList=491188 | | cardinality=21, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | PLAN FRAGMENT 2 | | | | PARTITION: HASH_PARTITIONED: c1[#0] | | | | HAS_COLO_PLAN_NODE: false | | | | STREAM DATA SINK | | EXCHANGE ID: 02 | | HASH_PARTITIONED: c2[#2] | | | | 1:VAGGREGATE (update serialize)(361) | | | STREAMING | | | output: partial_sum(c1[#0])[#3] | | | group by: c2[#1] | | | sortByGroupKey:false | | | cardinality=5 | | | distribute expr lists: c1[#0] | | | | | 0:VOlapScanNode(358) | | TABLE: tt.t2(t2), PREAGGREGATION: ON | | partitions=1/1 (t2) | | tablets=1/1, tabletList=491198 | | cardinality=10, avgRowSize=0.0, numNodes=1 | | pushAggOp=NONE | | | | | | Statistics | | planed with unknown column statistics | +------------------------------------------------------------+ 105 rows in set (0.06 sec) The Path to JOIN Decision After listening to Old Li's explanation, Xiao Zhang had an epiphany. JOIN optimization is not about simply choosing one solution; it's about making flexible decisions based on the actual situation: Joining a large table with a small table? Go all-in with Broadcast Join. Tables of similar size? Partition Shuffle is a safe bet. Left table bucketed appropriately? Bucket Shuffle shows its power. Joining tables in the same group? Colocate Join leads the way. Non-equi-JOIN? Nest Loop Join comes to the rescue. After putting these strategies into practice, Xiao Zhang compiled a set of JOIN optimization tips: Plan data distribution in advance to lay the foundation for high-performance JOIN operations.Leverage partition pruning to reduce the amount of data involved in JOINs.Design bucketing strategies wisely to create favorable conditions for Bucket Shuffle and Colocate Join.Configure parallelism and memory properly to maximize JOIN performance.Monitor resource usage closely to identify performance bottlenecks in a timely manner. By optimizing queries based on these insights, Xiao Zhang was able to reduce tasks that originally took hours to just a few minutes, earning high praise from his boss. The world of JOIN optimization is vast and complex; what we've explored today is just the tip of the iceberg. Choosing the right JOIN strategy for different scenarios is key to finding the optimal balance between performance and resource consumption.

I’ve been a web developer for years, but I haven’t touched Java in a long time — like, late-90s long. Back then, Java development felt cumbersome: lots of boilerplate and complex configurations. It was not exactly a pleasant experience for building simple web apps. So, when I recently started exploring Scala and the Play Framework, I was curious more than anything. Has the Java developer experience gotten better? Is it actually something I’d want to use today? Scala runs on the Java Virtual Machine, but it brings a more expressive, modern syntax to the table. It’s often used in backend development, especially when performance and scalability matter. Play is a web framework built for Scala, designed to be fast, reactive, and developer-friendly. And you use sbt, which is Scala’s build tool — roughly comparable to Maven or Gradle in the Java world. In this post, I’ll walk through setting up a basic Scala Play app, running it locally, and then deploying it to Heroku. My hope is to show you how to get your app running smoothly — without needing to know much about the JVM or how Play works under the hood. Introducing the Example App To keep things simple, I’m starting with an existing sample project: the Play Scala REST API example. It’s a small application that exposes a few endpoints for creating and retrieving blog posts. All the data is stored in memory, so there’s no database to configure — perfect for testing and deployment experiments. To make things easier for this walkthrough, I’ve cloned that sample repo and made a few tweaks to prep it for Heroku. You can follow along using my GitHub repo This isn’t a production-ready app, and that’s the point. It’s just robust enough to explore how Play works and see what running and deploying a Scala app actually feels like. Taking a Quick Look at the Code Before we deploy anything, let’s take a quick tour of the app itself. It’s a small codebase, but there are a few things worth pointing out — especially if you’re new to Scala or curious about how it compares to more traditional Java. I learned a lot about the ins and outs of the Play Framework for building an API by reading this basic explanation page. Here are some key points: Case Class to Model Resources For this REST API, the basic resource is the blog post, which has an ID, link, title, and body. The simplest way to model this is with a Scala case class that looks like this: Scala case class PostResource( id: String, link: String, title: String, body: String ) This happens at the top of app/v1/post/PostResourceHandler.scala. Routing Is Clean and Centralized The conf/routes file maps HTTP requests to a router in a format that’s easy to read and change. It feels closer to something like Express or Flask than an XML-based Java config. In our example, the file is just one line: Shell -> /v1/posts v1.post.PostRouter From there, the router at app/v1/post/PostRouter.scala defines how different requests within this path map to controller methods. Scala override def routes: Routes = { case GET(p"/") => controller.index case POST(p"/") => controller.process case GET(p"/$id") => controller.show(id) } This is pretty clear. A GET request to the path root takes us to the controller’s index method, while a POST request takes us to its process method. Meanwhile, a GET request with an included blog post id will take us to the show method and pass along the given id. Controllers Are Concise That brings us to our controller, at app/v1/post/PostController.scala. Each endpoint is a method that returns an Action, which works with the JSON result using Play’s built-in helpers. For example: Scala def show(id: String): Action[AnyContent] = PostAction.async { implicit request => logger.trace(s"show: id = $id") postResourceHandler.lookup(id).map { post => Ok(Json.toJson(post)) } } There’s very little boilerplate, so there is no need for separate interface declarations or verbose annotations. JSON Is Handled With Implicit Values Play uses implicit values to handle JSON serialization and deserialization. You define an implicit Format[PostResource] using Play JSON’s macros, and then Play just knows how to turn your objects into JSON and back again. No manual parsing or verbose configuration needed. We see this in app/v1/post/PostResourceHandler.scala: Scala object PostResource { /** * Mapping to read/write a PostResource out as a JSON value. */ implicit val format: Format[PostResource] = Json.format } Modern, Expressive Syntax As I dig around through the code, I see some of Scala’s more expressive features in action — things like map operations and pattern matching with match. The syntax looks new at first, but it quickly feels like a streamlined blend of Java and JavaScript. Running the App Locally Before deploying to the cloud, it’s always a good idea to make sure the app runs locally. This helps us catch any obvious issues and lets us poke around the API. On my machine, I make sure to have Java and sbt installed. Shell ~/project$ java --version openjdk 17.0.14 2025-01-21 OpenJDK Runtime Environment (build 17.0.14+7-Ubuntu-120.04) OpenJDK 64-Bit Server VM (build 17.0.14+7-Ubuntu-120.04, mixed mode, sharing) ~/project$ sbt --version sbt version in this project: 1.10.6 sbt script version: 1.10.6 Then, I run the following from my project root: Shell ~/project$ sbt run [info] welcome to sbt 1.10.6 (Ubuntu Java 17.0.14) [info] loading settings for project scala-api-heroku-build from plugins.sbt... [info] loading project definition from /home/alvin/repositories/devspotlight/heroku/scala/scala-api-heroku/project [info] loading settings for project root from build.sbt... [info] loading settings for project docs from build.sbt... [info] __ __ [info] \ \ ____ / /____ _ __ __ [info] \ \ / __ \ / // __ `// / / / [info] / / / /_/ // // /_/ // /_/ / [info] /_/ / .___//_/ \__,_/ \__, / [info] /_/ /____/ [info] [info] Version 3.0.6 running Java 17.0.14 [info] [info] Play is run entirely by the community. Please consider contributing and/or donating: [info] https://www.playframework.com/sponsors [info] --- (Running the application, auto-reloading is enabled) --- INFO p.c.s.PekkoHttpServer - Listening for HTTP on /[0:0:0:0:0:0:0:0]:9000 (Server started, use Enter to stop and go back to the console...) With my local server up and listening on port 9000, I open a new terminal and test the API by sending a request: Shell $ curl http://localhost:9000/v1/posts | jq [ { "id": "1", "link": "/v1/posts/1", "title": "title 1", "body": "blog post 1" }, { "id": "2", "link": "/v1/posts/2", "title": "title 2", "body": "blog post 2" }, { "id": "3", "link": "/v1/posts/3", "title": "title 3", "body": "blog post 3" }, { "id": "4", "link": "/v1/posts/4", "title": "title 4", "body": "blog post 4" }, { "id": "5", "link": "/v1/posts/5", "title": "title 5", "body": "blog post 5" } ] Nice. That was fast. Next, I want to play around with a few requests — retrieving and creating a blog post. Shell $ curl http://localhost:9000/v1/posts/3 | jq { "id": "3", "link": "/v1/posts/3", "title": "title 3", "body": "blog post 3" } $ curl -X POST \ --header "Content-type:application/json" \ --data '{ "title": "Just Another Blog Post", "body": "This is my blog post body." }' \ http://localhost:9000/v1/posts | jq { "id": "999", "link": "/v1/posts/999", "title": "Just Another Blog Post", "body": "This is my blog post body." } Preparing for Deployment to Heroku Ok, we’re ready to deploy our Scala Play app to Heroku. However, being new to Scala and Play, I’m predisposed to hitting a few speed bumps. I want to cover those so that you can steer clear of them in your development. Understanding the AllowHostsFilter When I run my app locally with sbt run, I have no problems sending curl requests and receiving responses. But as soon as I’m in the cloud, I’m using a Heroku app URL, not localhost. For security, Play has an AllowedHostsFilter enabled, which means you need to specify explicitly which hosts can access the app. I modify conf/application.conf to include the following block: Scala play.filters.hosts { allowed = [ "localhost:9000", ${?PLAY_ALLOWED_HOSTS} ] } This way, I can set the PLAY_ALLOWED_HOSTS config variable for my Heroku app, adding my Heroku app URL to the list of allowed hosts. Setting a Play Secret Play requires an application secret for security. It’s used for signing and encryption. You could set the application secret in conf/application.conf, but that hardcodes it into your repo, which isn’t a good practice. Instead, let’s set it at runtime based on an environment config variable. We add the following lines to conf/application.conf: Scala play.http.secret.key="changeme" play.http.secret.key=${?PLAY_SECRET} The first line sets a default secret key, while the second line sets the secret to come from our config variable, PLAY_SECRET, if it is set. sbt run Versus Running the Compiled Binary Lastly, I want to talk a little bit about sbt run. But first, let’s rewind: on my first attempt at deploying to Heroku, I thought I would just spin up my server by having Heroku execute sbt run. When I did that, my dyno kept crashing. I didn’t realize how memory-intensive this mode was. It’s meant for development, not production. During that first attempt, I turned on log-runtime-metrics for my Heroku app. My app was using over 800M in memory — way too much for my little Eco Dyno, which was only 512M. If I wanted to use sbt run, I would have needed a Performance-M dyno. That didn’t seem right. As I read Play’s documentation on deploying an application, I realized that I should run my app through the compiled binary that Play creates via another command, sbt stage. This version of my app is precompiled and much more memory efficient — down to around 180MB. Assuming sbt stage would run first, I just needed to modify my startup command to run the binary. Why I Went With Heroku I chose Heroku for running my Scala app because it removes a lot of the setup friction. I don’t need to manually configure a server or install dependencies. Heroku knows that this is a Scala app (by detecting the presence of project/build.properties and build.sbt files) and applies its buildpack for Scala. This means automatic handling of things like dependency resolution and compilation by running sbt stage. For someone just experimenting with Scala and Play, this kind of zero-config deployment is ideal. I can focus on understanding the framework and the codebase without getting sidetracked by infrastructure. Once I sorted out a few gotchas from early on—like the AllowedHostsFilter, Play secret, and startup command — deployment was quick and repeatable. Alright, let’s go! Deploying the App to Heroku For deployment, I have the Heroku CLI installed, and I authenticate with heroku login. My first step is to create a new Heroku app. Shell ~/project$ heroku apps:create scala-blog-rest-api Creating ⬢ scala-blog-rest-api... done https://scala-blog-rest-api-5d26d52bd1e4.herokuapp.com/ | https://git.heroku.com/scala-blog-rest-api.git Create the Procfile Next, I need to create the Procfile that tells Heroku how to spin up my app. In my project root, I create the file, which contains this one line: Shell web: target/universal/stage/bin/play-scala-rest-api-example -Dhttp.port=${PORT} Notice that the command for starting up this Heroku web process is not sbt run. Instead, we execute the binary found in target/universal/stage/bin/play-scala-rest-api-example. That’s the binary compiled after running sbt stage. Where does the play-scala-rest-api-example file name come from? This is set in build.sbt. Make sure that name is consistent between build.sbt and your Procfile. Also, we set the port dynamically at runtime to the value of the environment variable PORT. Heroku sets the PORT environment variable when it starts up our app, so we just need to let our app know what that port is. Specify the JDK Version Next, I create a one-line file called system.properties, specifying the JDK version I want to use for my app. Since my local machine uses openjdk 17, my system.properties file looks like this: Shell java.runtime.version=17 Set Heroku App Config Variables To make sure everything is in order, we need to set a few config variables for our app: PLAY_SECRET: A string of our choosing, required to be at least 256 bits.PLAY_ALLOWED_HOSTS: Our Heroku app URL, which is then included in the AllowedHostsFilter used in conf/application.conf. We set our config variables like this: Shell ~/project$ heroku config:set \ PLAY_SECRET='ga87Dd*A7$^SFsrpywMWiyyskeEb9&D$hG!ctWxrp^47HCYI' \ PLAY_ALLOWED_HOSTS='scala-blog-rest-api-5d26d52bd1e4.herokuapp.com' Setting PLAY_SECRET, PLAY_ALLOWED_HOSTS and restarting ⬢ scala-blog-rest-api... done, v2 PLAY_ALLOWED_HOSTS: scala-blog-rest-api-5d26d52bd1e4.herokuapp.com PLAY_SECRET: ga87Dd*A7$^SFsrpywMWiyyskeEb9&D$hG!ctWxrp^47HCYI Push Code to Heroku With our Procfile created and our config variables set, we’re ready to push our code to Heroku. Shell ~/project$ git push heroku main remote: Resolving deltas: 100% (14/14), done. remote: Updated 95 paths from 4dc4853 remote: Compressing source files... done. remote: Building source: remote: remote: -----> Building on the Heroku-24 stack remote: -----> Determining which buildpack to use for this app remote: -----> Play 2.x - Scala app detected remote: -----> Installing Azul Zulu OpenJDK 17.0.14 remote: -----> Priming Ivy cache... done remote: -----> Running: sbt compile stage remote: -----> Collecting dependency information remote: -----> Dropping ivy cache from the slug remote: -----> Dropping sbt boot dir from the slug remote: -----> Dropping sbt cache dir from the slug remote: -----> Dropping compilation artifacts from the slug remote: -----> Discovering process types remote: Procfile declares types -> web remote: remote: -----> Compressing... remote: Done: 128.4M remote: -----> Launching... remote: Released v5 remote: https://scala-blog-rest-api-5d26d52bd1e4.herokuapp.com/ deployed to Heroku remote: remote: Verifying deploy... done. To https://git.heroku.com/scala-blog-rest-api.git * [new branch] main -> main I love how I only need to create a Procfile and set some config variables, and then Heroku takes care of the rest of it for me. Time to Test All that’s left to do is test my Scala Play app with a few curl requests: Shell $ curl https://scala-blog-rest-api-5d26d52bd1e4.herokuapp.com/v1/posts \ | jq [ { "id": "1", "link": "/v1/posts/1", "title": "title 1", "body": "blog post 1" }, { "id": "2", "link": "/v1/posts/2", "title": "title 2", "body": "blog post 2" }, { "id": "3", "link": "/v1/posts/3", "title": "title 3", "body": "blog post 3" }, { "id": "4", "link": "/v1/posts/4", "title": "title 4", "body": "blog post 4" }, { "id": "5", "link": "/v1/posts/5", "title": "title 5", "body": "blog post 5" } ] $ curl https://scala-blog-rest-api-5d26d52bd1e4.herokuapp.com/v1/posts/4 \ | jq { "id": "4", "link": "/v1/posts/4", "title": "title 4", "body": "blog post 4" } $ curl -X POST \ --header "Content-type:application/json" \ --data '{"title":"My blog title","body":"this is my blog post"}' \ https://scala-blog-rest-api-5d26d52bd1e4.herokuapp.com/v1/posts \ | jq { "id": "999", "link": "/v1/posts/999", "title": "My blog title", "body": "this is my blog post" } The API server works. Deployment was smooth and simple. And I learned a lot about Scala and Play along the way. All in all, it’s been a good day. Wrapping Up Jumping into Scala after decades away from Java — and being more used to building web apps with JavaScript — wasn’t as jarring as I expected. Scala’s syntax felt concise and modern, and working with case classes and async controllers reminded me a bit of patterns I already use in Node.js. There’s still a learning curve with the tooling and how things are structured in Play, but overall, it felt approachable, not overwhelming. Deployment to the cloud had a few gotchas, especially around how Play handles allowed hosts, secrets, and memory usage in production. But once I understood how those pieces worked, getting the app live on Heroku was straightforward. With just a few config changes and a proper startup command, the process was clean and repeatable. For a first-time Scala build and deployment, I couldn’t have asked for a smoother experience. Are you ready to give it a try? Happy coding!

By Alvin Lee CORE

If you go to ChatGPT and ask it about Hyperlambda, at best, it will hallucinate wildly and generate something that resembles a bastardized mutation between Bash and Python. For the record, Claude is not any better here. The reason is that there's simply not enough Hyperlambda out there for the website scrapers from OpenAI and Anthropic to pick up its syntax. This was a sore thumb for me for a long time, so I realised I just had to do something about it. So, I started fine-tuning OpenAI on Hyperlambda. If I had known what a job this would be, I'd probably never have started. To teach Hyperlambda to the point where it does "most simple things correct," it took me 3,500 training snippets and about 550 validation snippets. Each snippet is, on average, about 20 lines of code; multiplied by 4,000, we're up to 80 KLOC of code. Notice that these are 80,000 lines of code with absolutely no purpose whatsoever besides being a requirement to teach an LLM a new programming language. So, as you can guess, it wasn't the most motivating job. If you're interested in seeing my training snippets, I've made them publicly available in a GitHub repo. When was the last time you created 80,000 lines of unusable code? KNOWING they would be completely useless too may I add as you started out... Lessons Learned The first and most important thing I can tell you is to completely ignore everything ChatGPT tells you about fine-tuning. The same is true for Claude. If you ask ChatGPT for help to set your hyperparameters, it will tell you that you should set the learning rate multiplier at 0.1 to 0.2. This might be OK if you're OpenAI, and you're scraping the entire web, but for teaching GPT-40-mini a new programming language, you have to turn this number up significantly! The defaults (1.8) are definitely the LR multiplier that gives me the best result. So, I chose the defaults for all hyperparameters, resulting in a medium-good LLM capable of "doing most simple things correctly." FYI, the above is 100% correct Hyperlambda, and for "simple requests" such as illustrated above, the model produces an accuracy of roughly 80 to 90%. Training Snippets If you want to fine-tune OpenAI, you will need to create structured training snippets. These are divided into "prompt" and "completion" values, where the prompt is an imagined question, and the completion is the expected result from the LLM. Below is a screenshot. Pssst... imagine writing 4,000 of these, and you get an idea of the magnitude of the work. However, don't go overboard with rephrasing your prompts. If you've got high-quality file-level comments explaining what a file does, that's probably good enough. The LLM will extrapolate these into an understanding that allows it to answer questions regardless of how your "prompts" are formatted. For me that implied I could start out with my system folder, which contains all core building blocks for the magic backend. Then I use the CRUD generator from Magic, which allows me to generate CRUD HTTP web API endpoints wrapping whatever database you want to wrap. This gave me roughly 1,000 high quality training snippets, leaving only 2,500 having to be implemented by me and my team. In addition, I created a custom GPT, which I used internally to take a single Hyperlambda file and generate multiple "variations" from it. This allowed me to take a snippet storing name, email, and phone into a CRM/contacts database, and generate five similar snippets saving other object types into the database. This helped the LLM to understand what arguments are, and which arguments are changing according to my table structure, etc. ChatGPT and generative AI were still beneficial during this process, albeit not as much as you'd typically think. The bulk of the snippets still had to be generated from a rich and varied code base, solving all sort of problems, from database access, to sending emails, to generating Fibonacci numbers and calculating even numbers using Magic's math libraries. Each training snippet should also contain a "system" message, such as for instance; "You are a Hyperlambda software development assistant. Your task is to generate and respond with Hyperlambda." Cost Training GPT-40-mini will cost you $2 to $5 per run with ~2MB of data in your training files and validation files. Training on GPT-40 (the big model) will cost you $50 to $100 per run. I did several runs with the big GPT-40, and the results were actually worse than GPT-40-mini. My suggestion is to use the mini model unless you've got very deep pockets. I spent a total of $1,500 to pull this through, which required hundreds of test runs and QA iterations, weeding out errors and correcting them as I went forward. Personally, I would not even consider trying to fine-tune GPT-40 (the big model) today for these reasons. Conclusion Fine-tuning an LLM is ridiculously hard work, and if you search for it, the only real information you'll find is rubbish blog posts about how to fine-tune the model to become "sarcastic". This takes 10 snippets. Training the LLM on a completely new domain requires thousands of snippets. A new programming language, well, below are the figures. 3,500 snippets become a "tolerable LLM doing more good than damage," which is where we're at now.5,000 snippets. Now, it is good.10,000 snippets. At this point, the LLM will understand your new language almost as well as it understands Python or JavaScript. If you want to fine-tune an LLM, I can tell you it is possible, but unless you've got a lot of existing data that can easily be transformed into OpenAI's "prompt and completion" structure, the job is massive. Personally, I spent one month writing Hyperlambda code for some 3 to 6 hours per day, 7 days a week — and at this point, it's still only "tolerable." Do I recommend it? Well, for 99% of your stuff simply adding RAG to the LLM is the correct solution. But for a programming language such as Hyperlambda it's not working, and you have to use fine tuning. I did not cover the structure of your training files in this article, but basically these are assumed to be JSONL files (each line in your file is a single JSON object). If you're interested in seeing this, you can find information about this on OpenAI's website. However, if you start down this path, I have to want you. It is an insane amount of work! And I doubt I'd want to do it again if I knew just how much work it actually was. But the idea is to fill in with more and more training snippets over time until I reach the sweet spot of 5,000 snippets, at which point we've got a 100% working backend AI-generator solution for creating backend code and APIs. If you want to try out the Hyperlambda generator, you can try it out in our AI chatbot. However, you have to phrase your question starting out with something like "Generate Hyperlambda that..." to put your backend requirements into the prompt.

By Thomas Hansen CORE

Are you looking for a lean, secure, and versatile Docker image for Apache JMeter to streamline your load testing of workflows? Look no further! Today, I’m excited to share a new Dockerfile I’ve crafted that delivers a lightweight Apache JMeter image without compromising on functionality. Whether you’re a developer, DevOps engineer, or QA professional, this image is designed to make your performance testing faster, easier, and more efficient. Why This JMeter Docker Image Stands Out This isn’t your average JMeter setup. Here’s what makes this Docker image special: Small and secure base: Built on Alpine Linux, known for its tiny footprint and security-first design.Powered by Liberica JDK: Uses bellsoft/liberica-openjdk-alpine, a free, open-source Java runtime optimized for modern deployments.Full JMeter functionality: Includes Apache JMeter for robust load testing, plus a handy script to install plugins as needed.Non-root execution: Runs as the jmeter user for enhanced security — no unnecessary root privileges here.Multi-architecture ready: Supports both amd64 (x86_64) and arm64 platforms, so it works seamlessly on everything from Intel-based servers to Apple M1/M2/M3 chips, AWS Graviton, or even a Raspberry Pi 4.Optimized size: Thanks to multi-stage builds, the image is slim — 51.78 MB compressed and 209.71 MB uncompressed. This combination of features ensures you get a lightweight, secure, and flexible JMeter environment that’s ready to tackle your testing needs. Getting Started: Basic Usage Building and running the image is a breeze. Here’s how to get going: Clone this repo or simply use docker pull qainsights/jmeter command. Build the Image In the directory containing the Dockerfile, run: Shell docker build -t my-jmeter-image . This creates an image tagged as my-jmeter-image. Run JMeter To execute a test, mount your test directory and specify your .jmx file: Shell docker run -v /path/to/your/test:/tests my-jmeter-image /tests/your-test.jmx Replace /path/to/your/test with the local path to your test files and your-test.jmx with your test script’s filename. Done! Multi-Architecture Support: Flexibility Across Platforms One of the standout features of this image is its multi-architecture support. Whether you’re running on a traditional x86_64 machine or an ARM64 device like an Apple Silicon Mac or AWS Graviton instance, this image has you covered. To build a multi-architecture image, use the provided build-multiarch.sh script: Shell # Make it executable chmod +x build-multiarch.sh # Check options ./build-multiarch.sh --help # Build and push to your registry ./build-multiarch.sh --name jmeter --tag 5.6.3 --registry your-registry/ --push The script is packed with options: -n, --name: Set the image name (e.g., jmeter).-t, --tag: Specify a version tag (e.g., 5.6.3).-r, --registry: Define your Docker registry.--push: Push the image to your registry after building. This makes it easy to deploy JMeter across diverse environments without worrying about compatibility. Advanced Usage: Customize Your Setup Installing JMeter Plugins Need specific plugins for your tests? You can add them directly in the Dockerfile. Just tweak the JMETER_PLUGINS argument with a comma-separated list: Shell ARG JMETER_PLUGINS="jpgc-udp=0.4,jpgc-dummy" Want a specific version? Use =version-number. Otherwise, it’ll grab the latest. Keeping It Lean The image stays lightweight, thanks to some clever optimizations: Multi-stage builds: Dependencies are separated from the runtime, reducing bloat.File cleanup: Strips out unnecessary docs and Windows batch files.Cache management: Clears temporary files and package caches.Minimal plugins: Only what you need, nothing more. These steps ensure you’re not hauling around extra weight, making your CI/CD pipelines or local runs faster. Why Size Matters At just 151.78 MB compressed, this image is significantly smaller than many JMeter setups, which often balloon past 500 MB. A smaller image means quicker downloads, faster deployments, and less resource overhead — perfect for scaling tests in the cloud or running locally on constrained hardware. This project is open to contributions. Have an idea to make it even better? Found a bug? Feel free to submit a Pull Request on the repository. Let’s build something awesome together! Final Thoughts This lightweight Apache JMeter Docker image is all about efficiency, security, and flexibility. Whether you’re load testing a web app, API, or microservice, it’s got the tools you need in a package that won’t weigh you down. Give it a spin, tweak it to your liking, and let me know how it works for you! Happy testing!

By NaveenKumar Namachivayam CORE

If you’ve been following this blog, you know I’ve had a running series on implementing PHP Zmanim, the library based on Kosher Java that simplifies the work needed to calculate times related to Jewish religious observance. (If you need a full explanation of what that is, check out the first blog in the series.) While the work of understanding and implementing PHP Zmanim has been fun for it’s own sake, I was working on a larger goal behind the scenes: A WordPress plugin that implemented the library in a way that non-programmers could use on their (WordPress-based) websites. Today, I’m thrilled to announce that the plugin is available for download. My wife is equally excited because it means I’ll (hopefully) go back to more consistently remembering to eat, bathe, and come to bed. Zmanim WP is free* and can be downloaded from within your WordPress environment (go to Plugins, Add a New Plugin, and search for “Zmanim WP” or even just “zmanim”). Like the upcoming movie The Thunderbolts*, the splat (or “asterisk” for youngsters and pedantic linguists) is doing a bit of heavy lifting. Zmanim WP is actually “freemium,” meaning that some features are available and free to you forever, and other capabilities are only open if you buy a license. Before you fire up the torches, grab anti-capitalist pitchforks, and head to my luxurious mansion in Cleveland with arson in your heart, note that all the proceeds are going to charity. The initial charity will be my synagogue because: The Rabbi is my cousin, and he’s an amazing guy…but more importantly,They’ve been alpha-testing this for over two years, and they've had to suffer all my false starts, mistakes, bad time calculations, and questions. They did so with patience, equanimity, and good humor. With all that said, I’d like to talk about how the plugin works overall, and then I’ll explain what’s in the free version versus what I’ve held back in exchange for some filthy lucre. How Zmanim WP Works In its simplest form, the day-to-day use (I’ll get to the initial setup in a minute) of this plugin involves shortcodes that might look like this: Plain Text [zman_sunset] or Plain Text [zman_shema] or Plain Text [zman_misheyakir] Basically, it’s a set of brackets ([...]) with the word “zman_” followed by a specific type of time. When a visitor comes to your live (WordPress-based) website and views a page, post, or widget with that shortcode, it will display the corresponding time. For example, I have it running on THIS website, with the location set to Disneyland in California (Lat: 33.8120918, Long: -117.9215545). By using the shortcode [ zman_sunset ], you will see TODAY’S sunset for that location, no matter which day you view this page: 5:46 pm. If you don’t include any other options, it will show the time for the current day, in a standard _hh:mm am/pm_ format. But the fact is that you CAN include options – a lot of them. For example (and this is just a sample. For the full list of options you should check out the Zmanim WP documentation on AdatoSystems.com) you can include a date option “tomorrow” will show the day after the one when the page is viewed. [ zman_alot date="tomorrow" ] (which, again, is set for “the happiest place on earth” would be 4:22 am)“sunday” (or monday, tuesday, etc) will show the next upcoming day of that name. So if it’s currently Monday, the following code will show sunset for the upcoming Wednesday: [ zman_sunset date="Wednesday" ] (which results in 5:46 pm)(an actual date) will show the time for the specified date. [ zman_alot date="2025-01-10" ] (that gives you 4:57 am) You can also include a time offset. Let’s say that Mincha starts 20 minutes before sunset every day. You could automatically display that time using the code: [ zman_sunset offset=-20 ] (resulting in 5:26 pm) You can also change the time formatting. This gets a little more into the weeds, as it leverages PHP’s built-in datetime formatting (Here’s a nice tutorial on how those formatting codes work.). Thus, if I wanted to get the date information along with the time (including seconds) for sunset, I could use this:[ zman_sunset dateformat="m-d-Y h:i:s a" ] (that would be: 02-26-2025 05:46:02 pm) And just to be super clear about things, you can use all of those codes together if you want. There are also some options that don’t work across the board. For example “lang” will let you specify hebrew or english for some output like the Torah Portion or the Date. But it wouldn’t work for sunset. Fall Back: Setting Up Zmanim WP I said I would get to this part. To be honest, the setup isn’t all THAT involved. Once you install the plugin, you’ll get a new menu in the WordPress Admin portal. Clicking on the top level menu takes you to the Main Options page: Here, you set the location (using latitude and longitude), the time zone, and a few other cosmetic elements. However, the main work of configuration happens on the Standard Zmanim Settings page. This is where you select the method of calculating each zman from dropdowns populated with a wide range of halachic opinions. Free Features The shortcodes that are available to all users are: Location and Reference zman_location – Displays the location, as defined on the admin page.zman_lat – Shows the latitude, as defined on the admin page.zman_long – Shows the longitude, as defined on the admin page.zman_tzone – Shows the Time Zone, as defined on the admin page.Special Dates and Dayszman_shaah – A halachic hour, or 1/12 of the available daylight, as calculated based on the shita selected in the drop-down on the admin page.zman_parsha – Provides the Torah reading for that weekzman_zmandate – Provides the datezman_chodesh – The day(s) for the indicated Rosh Chodesh. If the date indicated is “today” or “next”, text ONLY be visible if this week/next week is Rosh Chodesh.zman_molad – The day/time for the indicated Molad. If the date indicated is “today” or “next”, text ONLY be visible if this week/next week is the Molad. Standard Zmanim zman_sunrise – Netz haChachma (nautical sunrise, without elevation).zman_sunset – Shkia (nautical sunset, without elevation).zman_candles – The time for Shabbat candles, which is sunset/shkia minus the number of minutes indicated on the admin page.zman_alot – Alot haShachar (earliest time for tallit & tefillin).zman_misheyakir – Misheyakir (earliest time for tefillot).zman_shema – Sof Zman Kriat Shema (latest time to say Shema)zman_tefilla – Sof Zman Tefilla (latest time to say Shacharit)zman_gedola – Mincha Gedolazman_ketana – Mincha Ketanazman_plag – Plag haMinchazman_bain – B’ain haShmashot (time between sunset/shkia and tzeit haKochavim).zman_tzait – Tzait haKochavim (the time 3 stars are visible in the night sky). Paid Features The following shortcodes are only available in the paid version of the Zmanim WP plugin. Along with the shortcodes you see below, there are additional configuration screens to set up key options and elements of the codes. zman_earlyshkia -Displays the earliest shkia time for the week (Sunday – Thursday) in which the provided date occurs. Example 1: show earliest shkia for this week: 5:43 pmExample 2: show earliest shkia for next week: 5:49 pmzman_lateshkia – Displays the latest shkia time for the week (Sunday – Thursday) in which the provided date givenzman_fullyear* – as described in the “Full Year Display” section above, this displays a grid of times for a complete year.zman_early_shabbat_1* through zman_early_shabbat_4* – As described in the “Early Shabbat Options” section above, each of these will display one of two shortcode outputs for early/regular Shabbat.zman_misheyakirweekly – provide an array of times for misheyakir, either for this week or next week. Specific times are accessed by using the “daynum” option. The Last Word (for now) Going all the way back to my first “Time Data Series” post: “What time will afternoon prayers (Mincha) be this week?” is a deceptively complex question. It’s deceptive because “afternoon prayers” seems to be self-explanatory, but (as with so many things related to Jewish religious rules (halacha) there’s a vast amount of background, commentary, and specificity required. If answering this question has stymied you, the Zmanim WP plugin might just be able to help. For more details, including how to download, install, and upgrade to the paid version of the plugin, check out the Zmanim WP documentation on .

In one of my earlier posts, we discussed how to best find memory leaks and the reasons behind them. It's best to use a focused and modern tool like HeapHero to detect OutOfMemory errors and many other performance bottlenecks, as it can pinpoint the real culprits and suggest ways to optimize the usage of computing resources. Above, you can see that there are a few thousand objects of byte[], String, int[], etc. Let's discuss some ways of fixing OutOfMemoryErrors in Java. You can see which fixes are applicable in your scenario/code and apply them to save memory and run your programs better. Some of the ways discussed below may seem trivial to you, but remember that a few small corrections may add up to a big gain. 1. Use ByteBuffer from java.nio Instead of allocating large byte arrays that may be underutilized, it allows direct memory allocation (ByteBuffer.allocateDirect(size)) to reduce GC pressure and avoid unnecessary heap allocations. If you are dealing with dynamically growing byte arrays, avoid starting with an unnecessarily large array. Java ByteBuffer buffer = ByteBuffer.allocateDirect(1024); // Allocates 1KB in off-heap memory For example, instead of: Java ByteArrayOutputStream baos = new ByteArrayOutputStream(10000);// Too large Let Java handle the resizing when needed because JVMs and JREs are continuously improved to consume minimal resources and manage their resource cycle well. Java ByteArrayOutputStream baos = new ByteArrayOutputStream(); // Starts small and grows as needed Or provide a small initial size if you know the minimum required length beforehand. Java ByteArrayOutputStream baos = new ByteArrayOutputStream(2048); // Starts small and grows as needed 2. Use Streams to Process Data in Chunks Instead of reading an entire file into memory using a huge byte array, For example, don’t use: Java byte[] data = Files.readAllBytes(Path.of("myLargeFile.txt")); // Loads entire file into memory Instead, try this: Java try (BufferedInputStream bis = new BufferedInputStream(new FileInputStream("myLargeFile.txt")); ByteArrayOutputStream baos = new ByteArrayOutputStream()) { byte[] buffer = new byte[2048]; // Read in smaller chunks int bytesRead; while ((bytesRead = bis.read(buffer)) != -1) { baos.write(buffer, 0, bytesRead); } byte[] data = baos.toByteArray(); } 3. Using the New MemorySegment Interface in Java 21 You can access off-heap or on-heap memory with the Foreign Function and Memory (FFM) API efficiently. It introduces the concept of an Arena. You use an Arena to allocate a memory segment and control the lifecycle of native memory segments. SegmentAllocator from Project Panama (Java 21) allows better control over memory allocation. Instead of large heap-based arrays, allocate memory using MemorySegment, which reduces garbage collection overhead. When you use the try-with-resources, the Arena will be closed as soon as the try block ends, all memory segments associated with its scope are invalidated, and the memory regions backing them are deallocated. For example: Java import java.lang.foreign.*; String s = "My LARGE ......... LARGE string"; try (Arena arena = Arena.ofConfined()) { // Allocate off-heap memory MemorySegment nativeText = arena.allocateUtf8String(s); // Access off-heap memory for (int i = 0; i < s.length(); i++ ) { System.out.print((char)nativeText.get(ValueLayout.JAVA_BYTE, i)); } } // Off-heap memory is deallocated 4. Use Singleton Objects Wherever Possible Some utility classes need not be instantiated per request; there can just be a single static instance for the whole application/session. For example, Unmarshallers and Marshallers. Unmarshallers are a part of JAXB specification. They are used to convert XML data into Java objects to its XML representations. Similarly, Marshallers are used to convert Java objects into XML representations. These help in processing XML data in Java programs by mapping XML elements and attributes to Java fields and properties, using Java annotations. If you look closely into the JAXBContext class, you will see that it has static methodscreateUnmarshaller() / createMarshaller() methods, which is a clear indication that these could be better handled as a single static instance for the whole application/session. 5. Use Singleton Scope In Your Spring-Based Applications This way, the container creates a single instance of that bean for the whole application to share, wherever possible, keeping your business logic intact. If coding a web application, remember that the @application scope creates the bean instance for the lifecycle of a ServletContext, the @request scope creates a bean instance for a single HTTP request, while the session scope creates a bean instance for a particular HTTP session. Java @Bean @Scope("singleton") public SomeService someService() { return new SomeService(); } 6. Use Faster and Memory-Efficient Alternatives to Popular Collections Use Collections.singletonMap and Collections.singletonList (for Small Collections) For example, if you only need a single key-value pair or item, avoid using a full HashMap or ArrayList, which have overhead. Use ArrayDeque Instead of LinkedList LinkedList has high memory overhead due to storing node pointers (next/prev references). Instead, use ArrayDeque, which is faster and memory-efficient. Java import java.util.ArrayDeque; ArrayDeque<Integer> deque = new ArrayDeque<Integer>(); deque.add(22); deque.removeFirst(); Use Map.of() and List.of() (Immutable Collections) If you don't need to modify a collection, use immutable collections, which are compact and optimized. Java Map<String, Integer> map = Map.of("A", 1, "B", 2); List<String> list = List.of("X", "Y", "Z"); Use WeakHashMap for Caching If you store temporary data in a HashMap, it may never get garbage collected, so use WeakHashMap, which automatically removes entries when keys are no longer referenced. 7. Close Objects as Soon as Their Utility Finishes Unclosed network sockets, I/O streams, database connections, and database/network objects keep using memory and CPU resources, adding to the running cost of the application. We have discussed some time-tested ways of dealing with OutOfMemory errors in Java. Your comments are welcome.

Event-Driven Ansible enables real-time automation by automatically reacting to system events, logs, or alerts without manual intervention. This guide provides a step-by-step approach to setting up basic event-driven automation using Ansible Rulebooks and ansible.eda.range module. By the end of this tutorial, you will have created your first event-driven playbook that prints a hello message using ansible.eda.hello module. About the Module The ansible.eda.range module in Event-Driven Ansible (EDA) generates events within a specified numerical range. It is commonly used for testing event-driven workflows, simulating recurring triggers, and executing automation tasks at controlled intervals. Ansible Ansible can be installed using various methods, including package managers, source installation, and automation tools. I installed Ansible using YUM, which provides a simple and efficient setup process. Ansible Rulebook To install Ansible Rulebook, use pip by running the command pip install ansible-rulebook. This command installs all the required dependencies, enabling us to define and execute event-driven automation. To check the version of Ansible Rulebook, execute the command ansible-rulebook --version. This will display the current version installed on your system. Event-Driven Ansible Module To install the Event-Driven Ansible module, you can use the ansible-galaxy collection install ansible.eda command, which manages Ansible collections. This command allows you to easily download and install the required EDA collections from Ansible Galaxy. By using the ansible-galaxy collection install, you ensure that all necessary dependencies for event-driven automation are properly set up. First Rule Book YAML --- - name: First Rulebook hosts: locahost sources: - name: range ansible.eda.range: limit: 5 rules: - name: "Pring hello message" condition: event.i == 4 action: run_playbook: name: ansible.eda.hello To execute the above rulebook, use the command ansible-rulebook -i localhost -r first_rulebook.yml. This command specifies the inventory as localhost and runs the rulebook defined above. It triggers the event-driven automation defined within the rulebook for execution on your local machine. Conclusion Event-Driven Ansible allows for seamless automation by responding to real-time events, logs, or alerts, eliminating the need for manual intervention. This approach streamlines task execution based on dynamic triggers. In this process, you learned to set up Ansible, Rulebook, and Event-Driven Ansible collections. You then created a rulebook that prints a "hello" message and successfully executed it. Note: The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

By Binoj Melath Nalinakshan Nair CORE