DZone Spotlight

With a traditional lexical-based (or keyword-based) search, we will find documents that contain the exact word we searched for. Keyword search excels at precision but struggles with alternate phrasing or natural language. Semantic search addresses these limitations by capturing the intent behind documents and user queries. This is typically done by leveraging vector embeddings to map documents and queries into a high dimensional space and computing vector similarity to retrieve relevant results. For several systems, a single search method may fall short, resulting in incomplete information being shown to users. Combining the strengths of both search methods described above would allow us to deliver an effective search experience. Keyword-based search is supported well by systems like Elasticsearch and Apache Solr. Semantic search typically requires storage with a vector database and there exists a wide range of solutions. This post explains how we can support hybrid search involving both lexical and semantic search using a single and familiar storage system in Postgres. Let’s suppose we have the following table used by an application that allows users to search for products via keyword or natural language: SQL CREATE TABLE products ( id bigserial PRIMARY KEY, description VARCHAR(255), embedding vector(384) ); The description column contains a text/natural language description of the product. Postgres provides a default index on this column for full-text search, but we can also create a custom index to accelerate full-text search, which acts like an index for information retrieval. The embedding column stores vector (float) representations of product descriptions, capturing semantic meaning rather than words. The pgvector extension in Postgres brings with it the vector data type and vector similarity metrics — L2, cosine, and dot product distances. There are several ways of generating embeddings, for e.g., using word-level embeddings such as Word2Vec, Sentence/Document embeddings such as SBERT, or embeddings from transformer-based models such as the BERT model. For demonstration, we will insert the following data into the database: SQL INSERT INTO products (description) VALUES ('Organic Cotton Baby Onesie - Newborn Size, Blue'), ('Soft Crib Sheet for Newborn, Hypoallergenic'), ('Baby Monitor with Night Vision and Two-Way Audio'), ('Diaper Bag Backpack with Changing Pad - Unisex Design'), ('Stroller for Infants and Toddlers, Lightweight'), ('Car Seat for Newborn, Rear-Facing, Extra Safe'), ('Baby Food Maker, Steamer and Blender Combo'), ('Toddler Sippy Cup, Spill-Proof, BPA-Free'), ('Educational Toys for 6-Month-Old Baby, Colorful Blocks'), ('Baby Clothes Set - 3 Pack, Cotton, 0-3 Months'), ('High Chair for Baby, Adjustable Height, Easy to Clean'), ('Baby Carrier Wrap, Ergonomic Design for Newborns'), ('Nursing Pillow for Breastfeeding, Machine Washable Cover'), ('Baby Bath Tub, Non-Slip, for Newborn and Infant'), ('Baby Skincare Products - Lotion, Shampoo, Wash - Organic'); For embeddings, I used a SentenceTransformer model (aka SBERT) to generate embeddings and then stored them in the database. The following Python code demonstrates this: SQL descriptions = [product[1] for product in products] model = SentenceTransformer("all-MiniLM-L6-v2") embeddings = model.encode(descriptions) # Update the database with embeddings for i, product in enumerate(products): product_id = product[0] embedding = embeddings[i] # Convert to Python list # Construct the vector string representation embedding_str = str(embedding.tolist()) cur.execute("UPDATE products SET embedding = %s WHERE id = %s", (embedding_str, product_id)) # Commit changes and close connection conn.commit() Full-Text Search Postgres provides extensive out-of-the-box support for keyword search. We can write a query like the following for keyword-based retrieval: Let’s say we want to search for sleep accessories for a baby. We might search with the following query: SQL SELECT id, description FROM products WHERE description @@ to_tsquery('english', 'crib | baby | bed'); This returns the following product back: SQL "Soft Crib Sheet for Newborn, Hypoallergenic" Note: ts_query searches for lexemes/normalized keywords, so replacing newborn with newborns or babies also returns the same result The above is, of course, a simple example, and Postgres’s full-text search functionality allows us several customizations, e.g., skip certain words, process synonyms, use sophisticated parsing, etc., by overriding the default text search config. Although these queries will work without an index, most applications find this approach too slow, except perhaps for occasional ad-hoc searches. Practical use of text searching usually requires creating an index. The following code demonstrates how we can create a GIN index (Generalized Inverted Index) on the description column and use it for efficient search. SQL --Create a tsvector column (you can add this to your existing table) ALTER TABLE products ADD COLUMN description_tsv tsvector; --Update the tsvector column with indexed data from the description column UPDATE products SET description_tsv = to_tsvector('english', description); -- Create a GIN index on the tsvector column CREATE INDEX idx_products_description_tsv ON products USING gin(description_tsv); Semantic Search Example Let’s now try to execute a semantic search request for our query intent — "baby sleeping accessories." To do this, we compute the embedding (as above) and pick the most similar products by vector distance (in this case, cosine distance). The following code demonstrates this: Python # The query string query_string = 'baby sleeping accessories' # Generate embedding for the query string query_embedding = model.encode(query_string).tolist() # Construct the SQL query using the cosine similarity operator (<->) # Assuming you have an index that supports cosine similarity (e.g., ivfflat with vector_cosine_ops) sql_query = """ SELECT id, description, (embedding <-> %s::vector) as similarity FROM products ORDER BY similarity LIMIT 5; """ # Execute the query cur.execute(sql_query, (query_embedding,)) # Fetch and print the results results = cur.fetchall() for result in results: product_id, description, similarity = result print(f"ID: {product_id}, Description: {description}, Similarity: {similarity}") cur.close() conn.close() This gives us the following results: Plain Text ID: 12, Description: Baby Carrier Wrap, Ergonomic Design for Newborns, Similarity: 0.9956936200879117 ID: 2, Description: Soft Crib Sheet for Newborn, Hypoallergenic, Similarity: 1.0233573590998544 ID: 5, Description: Stroller for Infants and Toddlers, Lightweight, Similarity: 1.078171715208051 ID: 6, Description: Car Seat for Newborn, Rear-Facing, Extra Safe, Similarity: 1.08259154868697 ID: 3, Description: Baby Monitor with Night Vision and Two-Way Audio, Similarity: 1.0902734271784085 Along with each result, we also get back its similarity (lower is better for cosine similarity). As we can see we get a richer set of results with embedding search which nicely augment the keyword based search. By default pgvector performs exact nearest neighbor search which guarantees perfect recall. However, this approach is quite expensive as the size of the dataset increases. We can add an index that trades off recall for speed. One example is the IVFFlat (Inverted File with Flat Compression) index in Postgres, which works by dividing the vector space into clusters using k-means clustering. During a search, it identifies the clusters closest to the query vector and performs a linear scan within those selected clusters, calculating the exact distances between the query vector and the vectors in those clusters. The following code defines how such an index can be created: SQL CREATE INDEX ON products USING ivfflat (embedding vector_cosine_ops) WITH (lists = 100); lists indicates the number of clusters to create. vector_cosine_ops indicates the distance metric we are using (cosine, inner product, or Euclidean/L2) Fusion of Results The two methods described above excel in different scenarios and complement each other. Combining the results of both methods would result in robust search results. Reciprocal Rank Fusion is a method for combining multiple result sets with different relevance indicators into a single result set. RRF requires no tuning, and the different relevance indicators do not have to be related to each other to achieve high-quality results. The core of RRF is captured in its formula: Mathematica RRF(d) = (r R) 1 / k + r(d)) Where: - d is a document - R is the set of rankers (retrievers) - k is a constant (typically 60) - r(d) is the rank of document d in ranker r In our example, we’d do the following: Calculate the rank of each product in each result set by taking the inverse of its rank after adding a constant. This constant prevents top-ranked products from dominating the final score and allows lower-ranked products to contribute meaningfully.Sum rank reciprocals from all result sets to get the final RRF score of a product. For keyword search, Postgres provides a ranking function ts_rank (and some variants) which can be used as the rank of a product inside the result set. For semantic search, we can use the embedding distance to calculate the rank of a product in the result set. It can be implemented in SQL using CTEs for each search method and combining them at the end. Further, we could also use an ML model to rerank the results after combining. Due to its high computational cost, ML model-based reranking is applied after the initial retrieval, which reduces the result set to a small set of promising candidates. Conclusion With the components described above, we built an intelligent search pipeline that integrates: Full-text search for precise keyword matchingVector search for semantic matchingResult fusion for combining results and reranking using ML We accomplished this by using a single database system, where all the data is stored. By avoiding integration with separate search engines or databases, we eliminated the need for having multiple tech stacks and reduced system complexity.

The LLM can work with the knowledge it has from its training data. To extend the knowledge retrieval-augmented generation (RAG) can be used that retrieves relevant information from a vector database and adds it to the prompt context. To provide really up-to-date information, function calls can be used to request the current information (flight arrival times, for example) from the responsible system. That enables the LLM to answer questions that require current information for an accurate response. The AIDocumentLibraryChat has been extended to show how to use the function call API of Spring AI to call the OpenLibrary API. The REST API provides book information for authors, titles, and subjects. The response can be a text answer or an LLM-generated JSON response. For the JSON response, the Structured Output feature of Spring AI is used to map the JSON in Java objects. Architecture The request flow looks like this: The LLM gets the prompt with the user question.The LLM decides if it calls a function based on its descriptions.The LLM uses the function call response to generate the answer.The Spring AI formats the answer as JSON or text according to the request parameter. Implementation Backend To use the function calling feature, the LLM has to support it. The Llama 3.1 model with function calling support is used by the AIDocumentLibraryChat project. The properties file: Properties files # function calling spring.ai.ollama.chat.model=llama3.1:8b spring.ai.ollama.chat.options.num-ctx=65535 The Ollama model is set, and the context window is set to 64k because large JSON responses need a lot of tokens. The function is provided to Spring AI in the FunctionConfig class: Java @Configuration public class FunctionConfig { private final OpenLibraryClient openLibraryClient; public FunctionConfig(OpenLibraryClient openLibraryClient) { this.openLibraryClient = openLibraryClient; } @Bean @Description("Search for books by author, title or subject.") public Function<OpenLibraryClient.Request, OpenLibraryClient.Response> openLibraryClient() { return this.openLibraryClient::apply; } } First, the OpenLibraryClient gets injected. Then, a Spring Bean is defined with its annotation, and the @Description annotation that provides the context information for the LLM to decide if the function is used. Spring AI uses the OpenLibraryClient.Request for the call and the OpenLibraryClient.Response for the answer of the function. The method name openLibraryClient is used as a function name by Spring AI. The request/response definition for the openLibraryClient() is in the OpenLibraryClient: Java public interface OpenLibraryClient extends Function<OpenLibraryClient.Request, OpenLibraryClient.Response> { @JsonIgnoreProperties(ignoreUnknown = true) record Book(@JsonProperty(value= "author_name", required = false) List<String> authorName, @JsonProperty(value= "language", required = false) List<String> languages, @JsonProperty(value= "publish_date", required = false) List<String> publishDates, @JsonProperty(value= "publisher", required = false) List<String> publishers, String title, String type, @JsonProperty(value= "subject", required = false) List<String> subjects, @JsonProperty(value= "place", required = false) List<String> places, @JsonProperty(value= "time", required = false) List<String> times, @JsonProperty(value= "person", required = false) List<String> persons, @JsonProperty(value= "ratings_average", required = false) Double ratingsAverage) {} @JsonInclude(Include.NON_NULL) @JsonClassDescription("OpenLibrary API request") record Request(@JsonProperty(required=false, value="author") @JsonPropertyDescription("The book author") String author, @JsonProperty(required=false, value="title") @JsonPropertyDescription("The book title") String title, @JsonProperty(required=false, value="subject") @JsonPropertyDescription("The book subject") String subject) {} @JsonIgnoreProperties(ignoreUnknown = true) record Response(Long numFound, Long start, Boolean numFoundExact, List<Book> docs) {} } The annotation @JsonPropertyDescription is used by Spring AI to describe the function parameters for the LLM. The annotation is used on the request record and each of its parameters to enable the LLM to provide the right values for the function call. The response JSON is mapped in the response record by Spring and does not need any description. The FunctionService processes the user questions and provides the responses: Java @Service public class FunctionService { private static final Logger LOGGER = LoggerFactory .getLogger(FunctionService.class); private final ChatClient chatClient; @JsonPropertyOrder({ "title", "summary" }) public record JsonBook(String title, String summary) { } @JsonPropertyOrder({ "author", "books" }) public record JsonResult(String author, List<JsonBook> books) { } private final String promptStr = """ Make sure to have a parameter when calling a function. If no parameter is provided ask the user for the parameter. Create a summary for each book based on the function response subject. User Query: %s """; @Value("${spring.profiles.active:}") private String activeProfile; public FunctionService(Builder builder) { this.chatClient = builder.build(); } public FunctionResult functionCall(String question, ResultFormat resultFormat) { if (!this.activeProfile.contains("ollama")) { return new FunctionResult(" ", null); } FunctionResult result = switch (resultFormat) { case ResultFormat.Text -> this.functionCallText(question); case ResultFormat.Json -> this.functionCallJson(question); }; return result; } private FunctionResult functionCallText(String question) { var result = this.chatClient.prompt().user( this.promptStr + question).functions("openLibraryClient") .call().content(); return new FunctionResult(result, null); } private FunctionResult functionCallJson(String question) { var result = this.chatClient.prompt().user(this.promptStr + question).functions("openLibraryClient") .call().entity(new ParameterizedTypeReference<List<JsonResult>>() {}); return new FunctionResult(null, result); } } In the FunctionService are the records for the responses defined. Then, the prompt string is created, and the profiles are set in the activeProfile property. The constructor creates the chatClient property with its Builder. The functionCall(...) method has the user question and the result format as parameters. It checks for the ollama profile and then selects the method for the result format. The function call methods use the chatClient property to call the LLM with the available functions (multiple possible). The method name of the bean that provides the function is the function name, and they can be comma-separated. The response of the LLM can be either got with .content() as an answer string or with .Entity(...) as a JSON mapped in the provided classes. Then, the FunctionResult record is returned. Conclusion Spring AI provides an easy-to-use API for function calling that abstracts the hard parts of creating the function call and returning the response as JSON. Multiple functions can be provided to the ChatClient. The descriptions can be provided easily by annotation on the function method and on the request with its parameters. The JSON response can be created with just the .entity(...) method call. That enables the display of the result in a structured component like a tree. Spring AI is a very good framework for working with AI and enables all its users to work with LLMs easily. Frontend The frontend supports the request for a text response and a JSON response. The text response is displayed in the frontend. The JSON response enables the display in an Angular Material Tree Component. Response with a tree component: The component template looks like this: XML <mat-tree [dataSource]="dataSource" [treeControl]="treeControl" class="example-tree"> <mat-tree-node *matTreeNodeDef="let node" matTreeNodeToggle> <div class="tree-node"> <div> <span i18n="@@functionSearchTitle">Title</span>: {{ node.value1 } </div> <div> <span i18n="@@functionSearchSummary">Summary</span>: {{ node.value2 } </div> </div> </mat-tree-node> <mat-nested-tree-node *matTreeNodeDef="let node; when: hasChild"> <div class="mat-tree-node"> <button mat-icon-button matTreeNodeToggle> <mat-icon class="mat-icon-rtl-mirror"> {{ treeControl.isExpanded(node) ? "expand_more" : "chevron_right" } </mat-icon> </button> <span class="book-author" i18n="@@functionSearchAuthor"> Author</span> <span class="book-author">: {{ node.value1 }</span> </div> <div [class.example-tree-invisible]="!treeControl.isExpanded(node)" role="group"> <ng-container matTreeNodeOutlet></ng-container> </div> </mat-nested-tree-node> </mat-tree> The Angular Material Tree needs the dataSource, hasChild and the treeControl to work with. The dataSource contains a tree structure of objects with the values that need to be displayed. The hasChild checks if the tree node has children that can be opened. The treeControl controls the opening and closing of the tree nodes. The <mat-tree-node ... contains the tree leaf that displays the title and summary of the book. The mat-nested-tree-node ... is the base tree node that displays the author's name. The treeControl toggles the icon and shows the tree leaf. The tree leaf is shown in the <ng-container matTreeNodeOutlet> component. The component class looks like this: TypeScript export class FunctionSearchComponent { ... protected treeControl = new NestedTreeControl<TreeNode>( (node) => node.children ); protected dataSource = new MatTreeNestedDataSource<TreeNode>(); protected responseJson = [{ value1: "", value2: "" } as TreeNode]; ... protected hasChild = (_: number, node: TreeNode) => !!node.children && node.children.length > 0; ... protected search(): void { this.searching = true; this.dataSource.data = []; const startDate = new Date(); this.repeatSub?.unsubscribe(); this.repeatSub = interval(100).pipe(map(() => new Date()), takeUntilDestroyed(this.destroyRef)) .subscribe((newDate) => (this.msWorking = newDate.getTime() - startDate.getTime())); this.functionSearchService .postLibraryFunction({question: this.searchValueControl.value, resultFormat: this.resultFormatControl.value} as FunctionSearch) .pipe(tap(() => this.repeatSub?.unsubscribe()), takeUntilDestroyed(this.destroyRef), tap(() => (this.searching = false))) .subscribe(value => this.resultFormatControl.value === this.resultFormats[0] ? this.responseText = value.result || '' : this.responseJson = this.addToDataSource(this.mapResult( value.jsonResult || [{ author: "", books: [] }] as JsonResult[]))); } ... private addToDataSource(treeNodes: TreeNode[]): TreeNode[] { this.dataSource.data = treeNodes; return treeNodes; } ... private mapResult(jsonResults: JsonResult[]): TreeNode[] { const createChildren = (books: JsonBook[]) => books.map(value => ({ value1: value.title, value2: value.summary } as TreeNode)); const rootNode = jsonResults.map(myValue => ({ value1: myValue.author, value2: "", children: createChildren(myValue.books) } as TreeNode)); return rootNode; } ... } The Angular FunctionSearchComponent defines the treeControl, dataSource, and the hasChild for the tree component. The search() method first creates a 100ms interval to display the time the LLM needs to respond. The interval gets stopped when the response has been received. Then, the function postLibraryFunction(...) is used to request the response from the backend/AI. The .subscribe(...) function is called when the result is received and maps the result with the methods addToDataSource(...) and mapResult(...) into the dataSource of the tree component. Conclusion The Angular Material Tree component is easy to use for the functionality it provides. The Spring AI structured output feature enables the display of the response in the tree component. That makes the AI results much more useful than just text answers. Bigger results can be displayed in a structured manner that would be otherwise a lengthy text. A Hint at the End The Angular Material Tree component creates all leafs at creation time. With a large tree with costly components in the leafs like Angular Material Tables the tree can take seconds to render. To avoid this treeControl.isExpanded(node) can be used with @if to render the tree leaf content at the time it is expanded. Then the tree renders fast, and the tree leafs are rendered fast, too.

Building quality software is only possible with quality tests. Whether you write test scripts for QA engineers or build automated tests, tests help to ensure that your applications continue to function as they grow and evolve. However, using automated testing to verify correct behavior can be challenging if your application generates visual artifacts, such as QR codes. Granted, you can write unit tests to ensure your code for generating QR codes does what it should; however, the danger is that your tests will be too tightly coupled to your application code. You’ll need to update your tests whenever you change the application. I’ve written about Tricentis Tosca before but recently uncovered one of its newer features. Among the suite of tools in the product is the visual verification of QR codes and barcodes. Instead of writing code to decode what my app just encoded, I can just point Tosca at my app and assert what should be in the QR code, setting aside the implementation details of how the tests work. Yes, this is how QR code testing ought to work. Let’s take a look at an example of how this testing works. Demo Application Before we can test QR code generation, we need to build an app that generates QR codes. Fortunately, we can use an open source QR code generator library. This repo includes examples for implementing QR code generation in several different languages, including Java, Python, TypeScript, C, and C++. I decided to go with the Rust implementation. When I built the Rust example, it had nice output in my console, so I went with that. I also added some code to the example to write the first QR code to an SVG file for testing later. Note that the QR code contains the string "Hello, world!" This is what we’ll test for. Testing QR Codes With Tosca Now that we have an app to work with, let’s look at how to test it with Tricentis Tosca. As a reminder, Tricentis Tosca is codeless, AI-powered testing for your applications. It offers both cloud and local agents for running application tests. In a recent release, it introduced new support for testing QR codes. Once my Rust code generated the QR code, I put the QR code in a PDF (the easiest way to do that is to open it and print it to PDF) and then put that PDF file in a location accessible to Tosca. I decided to use the new test builder in Tosca Cloud to build this test since I just wanted to try it out with a simple test case. This was as simple as logging into my Tosca Cloud account and starting a new test suite. Since I’d already connected the machine to Tosca Cloud and the VM is where I’ve set up my license to use Tosca, Tosca Cloud could run the test on the node as a personal agent. To configure the test, I simply dragged the module from Tosca’s module library for "Reading a PDF QR/Barcode" and pointed it at the PDF file. For the value field, I entered what should be in the file. That's it! Here’s how it looks in Tosca Cloud: Tosca also has a Web QR/Barcode validator, but using it requires me to publish an app with a URL that a test agent can access. After setting everything up, I clicked the Run button. Tosca Cloud starts an application locally, running on my test machine and linking the test case I’ve built. This allows me to skip much of the setup necessary for a more robust test suite. The test runs, and the test case executes. Tosca provides access to the results: To see the full output of that last step, we can click on it to see more details. Now, we have a passing test for our application based on the value encoded in the QR code. We don’t need to build a QR code decoder ourselves. Tosca handles decoding for us! This way, we can validate that our application behaves as expected. Let’s verify this by modifying what the test searches for in the generated QR code. We’ve updated the test to search for "Hello, Tosca!" instead of "Hello, world!". At this point, our updated test ought to fail. We run the test and look at the result. The test case fails, as expected. The details in the Verify step give us helpful information: As we can see, the QR code does not contain the string that the test is looking for. With the application currently encoding "Hello, world!" our test fails. Let’s fix our code to match our test expectations. We run our test again. The test passes, and the test case details confirm why. Now, as we change our application code or test requirements, we are confident that our QR codes generate according to spec. QR Testing for the Win! While this demonstration is somewhat simple, it shows the convenience of having a testing tool for QR code decoding. Additionally, the tests I’ve built for this feature are completely separate from the library I’ve used to implement the encoding functionality. This means that in the future, I can swap out the library without changing the tests at all, and my tests will tell me if the library change was seamless. Have a really great day!

By John Vester CORE

Prometheus is a powerful monitoring tool that provides extensive metrics and insights into your infrastructure and applications, especially in k8s and OCP (enterprise k8s). While crafting PromQL (Prometheus Query Language) expressions, ensuring accuracy and compatibility is essential, especially when comparing metrics or calculating thresholds. In this article, we will explore how to count worker nodes and track changes in resources effectively using PromQL. Counting Worker Nodes in PromQL To get the number of worker nodes in your Kubernetes cluster, the kube_node_info metric is often used. However, this metric includes all nodes, such as master, infra, and logging nodes, in addition to worker nodes. To filter only the worker nodes, you can refine your query using label matchers. Here is a query to count only worker nodes: Plain Text count(kube_node_info{node=~".*worker.*"}) Explanation kube_node_info is the metric that provides information about all nodes.{node=~".*worker.*"} filters nodes whose names contain the substring "worker."count() calculates the total number of matching nodes. This query ensures that only worker nodes are counted, which is often required for scaling metrics or thresholds in PromQL. Tracking Changes in Resource Usage A common use case in Kubernetes monitoring is tracking the change in the number of pods over time. For example, you might want to detect if pods have increased significantly within the last 30 minutes. Combining this with the worker node count allows you to set thresholds that scale with your cluster's size. Consider the following query: Plain Text max(apiserver_storage_objects{resource="pods"}) - max(apiserver_storage_objects{resource="pods"} offset 30m) > (20 * count(kube_node_info{node=~".*worker.*"})) Breakdown 1. Left-Hand Side max(apiserver_storage_objects{resource="pods"}) gets the maximum number of pods currently in the cluster.max(apiserver_storage_objects{resource="pods"} offset 30m) retrieves the maximum number of pods 30 minutes ago.Subtraction changes the number of pods over the last 30 minutes. 2. Right-Hand Side count(kube_node_info{node=~".*worker.*"}) counts the number of worker nodes.Multiplying this by 20 sets a dynamic threshold based on the number of worker nodes. 3. Comparison The query checks if the change in pod count exceeds the calculated threshold. Addressing Syntax Issues in PromQL While crafting PromQL expressions, syntax errors or mismatched types can lead to unexpected results. In the example above, the left-hand side of the query might return multiple time series, while the right-hand side is a scalar. To ensure compatibility, you can wrap the left-hand side in a max() function to reduce it to a scalar: Plain Text max(max(apiserver_storage_objects{resource="pods"}) - max(apiserver_storage_objects{resource="pods"} offset 30m)) > (20 * count(kube_node_info{node=~".*worker.*"})) Why Use max()? The max() function ensures that the result of the subtraction is a single scalar value, making it compatible with the right-hand side. General Best Practices Understand your metrics: Always familiarize yourself with the metrics you are querying. Use label_values() or the Prometheus UI to inspect available labels and their values.Test incrementally: Start with smaller queries and validate their results before building complex expressions.Ensure scalar compatibility: When comparing values, ensure both sides of the comparison are scalars. Use aggregation functions like max(), sum(), or avg() as needed.Dynamic thresholds: Use cluster-specific metrics (e.g., node count) to set thresholds that scale dynamically with your infrastructure. Conclusion PromQL is a powerful tool, but crafting accurate and efficient queries requires careful attention to detail. By using refined expressions like count(kube_node_info{node=~".*worker.*"}) to count worker nodes and dynamic thresholds based on cluster size, you can create robust monitoring solutions that adapt to your environment. Always test and validate your queries to ensure they provide meaningful insights. Feel free to use the examples and best practices discussed here to enhance your monitoring setup and stay ahead of potential issues in your Kubernetes cluster.

Elasticsearch and OpenSearch are powerful tools for handling search and analytics workloads, offering scalability, real-time capabilities, and a rich ecosystem of plugins and integrations. Elasticsearch is widely used for full-text search, log monitoring, and data visualization across industries due to its mature ecosystem. OpenSearch, a community-driven fork of Elasticsearch, provides a fully open-source alternative with many of the same capabilities, making it an excellent choice for organizations prioritizing open-source principles and cost efficiency. Migration to OpenSearch should be considered if you are using Elasticsearch versions up to 7.10 and want to avoid licensing restrictions introduced with Elasticsearch's SSPL license. It is also ideal for those seeking continued access to an open-source ecosystem while maintaining compatibility with existing Elasticsearch APIs and tools. Organizations with a focus on community-driven innovation, transparent governance, or cost control will find OpenSearch a compelling option. History Elasticsearch, initially developed by Shay Banon in 2010, emerged as a powerful open-source search and analytics engine built on Apache Lucene. It quickly gained popularity for its scalability, distributed nature, and robust capabilities in full-text search, log analysis, and real-time data processing. Over the years, Elasticsearch became part of the Elastic Stack (formerly ELK Stack), integrating with Kibana, Logstash, and Beats to provide end-to-end data management solutions. However, a significant shift occurred in 2021 when Elastic transitioned Elasticsearch and Kibana to a more restrictive SSPL license. In response, AWS and the open-source community forked Elasticsearch 7.10 and Kibana to create OpenSearch, adhering to the Apache 2.0 license. OpenSearch has since evolved as a community-driven project, ensuring a truly open-source alternative with comparable features and ongoing development tailored for search, observability, and analytics use cases. Why Migrate to OpenSearch? 1. Open Source Commitment OpenSearch adheres to the Apache 2.0 license, ensuring true open-source accessibility. In contrast, Elasticsearch's transition to a more restrictive SSPL license has raised concerns about vendor lock-in and diminished community-driven contributions. 2. Cost Efficiency OpenSearch eliminates potential licensing fees associated with Elasticsearch's newer versions, making it an attractive choice for organizations seeking cost-effective solutions without compromising on capabilities. 3. Compatibility OpenSearch maintains compatibility with Elasticsearch versions up to 7.10, including many of the same APIs and tools. This ensures a smooth migration with minimal disruption to existing applications and workflows. 4. Active Development and Support Backed by AWS and an active community, OpenSearch receives consistent updates, feature enhancements, and security patches. Its open governance model fosters innovation and collaboration, ensuring the platform evolves to meet user needs. 5. Customizable and Flexible OpenSearch allows for greater customization and flexibility compared to proprietary systems, enabling organizations to tailor their deployments to specific use cases without constraints imposed by licensing terms. 6. Evolving Ecosystem OpenSearch offers OpenSearch Dashboards (a Kibana alternative) and plugins tailored for observability, log analytics, and full-text search. These tools expand its usability across domains while ensuring continued alignment with user needs. When to Migrate Licensing concerns: If you wish to avoid SSPL licensing restrictions introduced by Elastic after version 7.10.Budgetary constraints: To minimize costs associated with commercial licensing while retaining a powerful search and analytics engine.Future-proofing: To adopt a platform with a transparent development roadmap and strong community backing.Feature parity: When using features supported in Elasticsearch 7.10 or earlier, as these are fully compatible with OpenSearch.Customization needs: When greater flexibility, open governance, or community-led innovations are critical to your organization’s goals. Migrating to OpenSearch ensures you maintain a robust, open-source-driven platform while avoiding potential restrictions and costs associated with Elasticsearch’s licensing model. Pre-Migration Checklist Before migrating from Elasticsearch to OpenSearch, follow this checklist to ensure a smooth and successful transition: 1. Assess Version Compatibility Verify that your Elasticsearch version is compatible with OpenSearch. OpenSearch supports Elasticsearch versions up to 7.10.Review any API or plugin dependencies to ensure they are supported in OpenSearch. 2. Evaluate Use of Proprietary Features Identify any proprietary features or plugins (e.g., Elastic's machine learning features) that may not have equivalents in OpenSearch.Assess whether third-party tools or extensions used in your Elasticsearch cluster will be impacted. 3. Backup Your Data Create a full backup of your Elasticsearch indices using the snapshot API to avoid any potential data loss: Shell PUT /_snapshot/backup_repo/snapshot_1?wait_for_completion=true Ensure backups are stored in a secure and accessible location for restoration. 4. Review Cluster Configurations Document your current Elasticsearch cluster settings, including node configurations, shard allocations, and index templates.Compare these settings with OpenSearch to identify any required adjustments. 5. Test in a Staging Environment Set up a staging environment to simulate the migration process.Restore data snapshots in the OpenSearch staging cluster to validate compatibility and functionality.Test your applications, queries, and workflows in the staging environment to detect issues early. 6. Check API and Query Compatibility Review the Elasticsearch APIs and query syntax used in your application. OpenSearch maintains most API compatibility, but slight differences may exist.Use OpenSearch’s API compatibility mode for smoother transitions. 7. Update Applications and Clients Replace Elasticsearch client libraries with OpenSearch-compatible libraries (e.g., opensearch-py for Python or OpenSearch Java Client).Test client integration to ensure applications interact correctly with the OpenSearch cluster. 8. Verify Plugin Support Ensure that any plugins used in Elasticsearch (e.g., analysis, security, or monitoring plugins) are available or have alternatives in OpenSearch.Identify OpenSearch-specific plugins that may enhance your cluster's functionality. 9. Inform Stakeholders Communicate the migration plan, timeline, and expected downtime (if any) to all relevant stakeholders.Ensure teams responsible for applications, infrastructure, and data are prepared for the migration. 10. Plan for Rollback Develop a rollback plan in case issues arise during the migration. This plan should include steps to restore the original Elasticsearch cluster and data from backups. 11. Monitor Resources Ensure your infrastructure can support the migration process, including disk space for snapshots and sufficient cluster capacity for restoration. By completing this checklist, you can minimize risks, identify potential challenges, and ensure a successful migration from Elasticsearch to OpenSearch. Step-by-Step Migration Guide 1. Install OpenSearch Download the appropriate version of OpenSearch from opensearch.org.Set up OpenSearch nodes using the official documentation, ensuring similar cluster configurations to your existing Elasticsearch setup. 2. Export Data from Elasticsearch Use the snapshot APIto create a backup of your Elasticsearch indices: Shell PUT /_snapshot/backup_repo/snapshot_1?wait_for_completion=true Ensure that the snapshot is stored in a repository accessible to OpenSearch. 3. Import Data into OpenSearch Register the snapshot repository in OpenSearch: Shell PUT /_snapshot/backup_repo { "type": "fs", "settings": { "location": "path_to_backup", "compress": true } } Restore the snapshot to OpenSearch: Shell POST /_snapshot/backup_repo/snapshot_1/_restore 4. Update Applications and Clients Update your application’s Elasticsearch client libraries to compatible OpenSearch clients, such as the OpenSearch Python Client (opensearch-py) or Java Client.Replace Elasticsearch endpoints in your application configuration with OpenSearch endpoints. 5. Validate Data and Queries Verify that all data has been restored successfully.Test queries, index operations, and application workflows to ensure everything behaves as expected. 6. Monitor and Optimize Use OpenSearch Dashboards (formerly Kibana) to monitor cluster health and performance.Enable security features like encryption, authentication, and role-based access controls if required. Post-Migration Considerations 1. Plugins and Features If you rely on Elasticsearch plugins, verify their availability or find OpenSearch alternatives. 2. Performance Tuning Optimize OpenSearch cluster settings to match your workload requirements.Leverage OpenSearch-specific features, such as ultra-warm storage, for cost-efficient data retention. 3. Community Engagement Join the OpenSearch community for support and updates.Monitor release notes to stay informed about new features and improvements. Challenges and Tips for Migrating from Elasticsearch to OpenSearch 1. Plugin Compatibility Challenge Some Elasticsearch plugins, especially proprietary ones, may not have direct equivalents in OpenSearch. Tips Audit your current Elasticsearch plugins and identify dependencies.Research OpenSearch’s plugin ecosystem or alternative open-source tools to replace missing features.Consider whether OpenSearch’s built-in capabilities, such as OpenSearch Dashboards, meet your needs. 2. API Differences Challenge While OpenSearch maintains compatibility with Elasticsearch APIs up to version 7.10, minor differences or deprecated endpoints may impact functionality. Tips Use OpenSearch’s API compatibility mode to test and adapt APIs gradually.Review API documentation and replace deprecated endpoints with recommended alternatives. 3. Data Migration Challenge Migrating large datasets can be time-consuming and prone to errors, especially if there are format or schema differences. Tips Use the snapshot and restore approach for efficient data transfer.Test the restoration process in a staging environment to ensure data integrity.Validate data post-migration by running key queries to confirm consistency. 4. Performance Tuning Challenge OpenSearch and Elasticsearch may have differences in cluster configurations and performance tuning, potentially leading to suboptimal performance post-migration. Tips Monitor cluster performance using OpenSearch Dashboards or other monitoring tools. Adjust shard sizes, indexing strategies, and resource allocation to optimize cluster performance. 5. Client and Application Integration Challenge Applications using Elasticsearch client libraries may require updates to work with OpenSearch. Tips Replace Elasticsearch clients with OpenSearch-compatible versions, such as opensearch-py (Python) or the OpenSearch Java Client.Test application workflows and query execution to ensure smooth integration. 6. Limited Features in OpenSearch Challenge Certain proprietary Elasticsearch features (e.g., machine learning jobs, Elastic Security) are not available in OpenSearch. Tips Identify critical features missing in OpenSearch and determine their importance to your use case.Explore third-party or open-source alternatives to replace unavailable features. 7. Training and Familiarity Challenge Teams familiar with Elasticsearch may face a learning curve when transitioning to OpenSearch, especially for cluster management and new features. Tips Provide training and documentation to familiarize your team with OpenSearch’s tools and workflows.Leverage OpenSearch’s active community and forums for additional support. 8. Real-Time Data and Downtime Challenge For real-time systems, ensuring minimal downtime during migration can be difficult. Tips Plan the migration during low-traffic periods.Use a blue-green deployment strategy to switch seamlessly between clusters.Sync new data into OpenSearch using tools like Logstash or Beats during the migration window. 9. Scalability and Future Growth Challenge Ensuring the new OpenSearch cluster can handle future growth and scalability requirements. Tips Plan for scalability by designing a cluster architecture that supports horizontal scaling.Use OpenSearch’s distributed architecture to optimize resource usage. 10. Community Support Challenge While OpenSearch has a growing community, some advanced issues may lack extensive documentation or third-party solutions. Tips Engage with the OpenSearch community via forums and GitHub for troubleshooting.Regularly monitor OpenSearch updates and contribute to the community for better insights. By anticipating these challenges and following these tips, organizations can navigate the migration process effectively, ensuring a seamless transition while maintaining search and analytics performance. Conclusion Migrating from Elasticsearch to OpenSearch is a strategic decision for organizations seeking to align with open-source principles, reduce costs, and maintain compatibility with established search and analytics workflows. While the migration process presents challenges, such as plugin compatibility, API differences, and data migration complexities, these can be effectively managed through careful planning, thorough testing, and leveraging the vibrant OpenSearch community.

While working on the series of tutorial blogs for GET, POST, PUT, PATCH, and DELETE requests for API Automation using Playwright Java. I noticed that there is no logging method provided by the Playwright Java framework to log the requests and responses. In the REST-assured framework, we have the log().all() method available that is used for logging the request as well as the response. However, Playwright does not provide any such method. However, Playwright offers a text() method in the APIResponse interface that could be well used to extract the response text. Playwright currently does not have the feature to access the request body and request headers while performing API Testing. The issue is already raised on GitHub for this feature, please add an upvote to this issue so this feature gets implemented soon in the framework. In this blog, we will learn how to extract the response and create a custom logger to log the response of the API tests using Playwright Java. How to Log Response Details in Playwright Java Before we begin with actual coding and implementation for the logger, let’s discuss the dependencies, configuration, and setup required for logging the response details. Getting Started As we are working with Playwright Java using Maven, we will use the Log4J2 Maven dependency to log the response details. The dependency for Jackson Databind will be used for parsing the JSON response. XML <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-api</artifactId> <version>${log4j-api-version}</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-core</artifactId> <version>${log4j-core-version}</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>${jackson-databind-version}</version> </dependency> As a best practice, the versions of these dependencies will be added in the properties block as it allows users to easily check and update the newer version of dependencies in the project. XML <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <log4j-core-version>2.24.1</log4j-core-version> <log4j-api-version>2.24.1</log4j-api-version> <jackson-databind-version>2.18.0</jackson-databind-version> </properties> The next step is to create a log4j2.xml file in the src/main/resources folder. This file stores the configuration for logs, such as log level, where the logs should be printed — to the console or to the file, the pattern of the log layout, etc. XML <?xml version="1.0" encoding="UTF-8"?> <Configuration status="INFO"> <Appenders> <Console name="LogToConsole" target="SYSTEM_OUT"> <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/> </Console> </Appenders> <Loggers> <Logger name="io.github.mfaisalkhatri" level="info" additivity="false"> <AppenderRef ref="LogToConsole"/> </Logger> <Root level="error"> <AppenderRef ref="LogToConsole"/> </Root> </Loggers> <Loggers> <Logger name="io.github.mfaisalkhatri" level="trace" additivity="false"> <AppenderRef ref="LogToConsole"/> </Logger> <Root level="error"> <AppenderRef ref="LogToConsole"/> </Root> </Loggers> </Configuration> The <Appenders> section contains the information related to log printing and its pattern format to print. The <Loggers> section contains the log-level details and how it should be printed. There can be multiple blocks of <Loggers> in the file, each for different log levels, such as “info,” “debug,” “trace,” etc. Implementing the Custom Logger A new Java class Logger is created to implement the methods for logging the response details. Java public class Logger { private final APIResponse response; private final org.apache.logging.log4j.Logger log; public Logger (final APIResponse response) { this.response = response; this.log = LogManager.getLogger (getClass ()); } //... } This class has the APIResponse interface of Playwright and the Logger interface from Log4j declared at the class level to ensure that we can reuse it in further methods in the same class and avoid duplicate code lines. The constructor of the Logger class is used for creating objects of the implementing classes. The APIResponse interface is added as a parameter as we need the response object to be supplied to this class for logging the respective details. The logResponseDetails() method implements the function to log all the response details. Java public void logResponseDetails () { String responseBody = this.response.text (); this.log.info ("Logging Response Details....\n responseHeaders: {}, \nstatusCode: {},", this.response.headers (), this.response.status ()); this.log.info ("\n Response body: {}", prettyPrintJson (responseBody)); this.log.info ("End of Logs!"); } The responseBody variable will store the response received after executing the API. The next line of code will print the response details, Headers, and Status Code. As the response returned is not pretty printed, meaning the JSON format is shown in String in multiple lines wrapped up, this makes the logs look untidy. Hence, we have created a prettyPrintJson() method that consumes the response in String format and returns it in pretty format. Java private String prettyPrintJson (final String text) { if (StringUtils.isNotBlank (text) && StringUtils.isNotEmpty (text)) { try { final ObjectMapper objectMapper = new ObjectMapper (); final Object jsonObject = objectMapper.readValue (text, Object.class); return objectMapper.writerWithDefaultPrettyPrinter () .writeValueAsString (jsonObject); } catch (final JsonProcessingException e) { this.log.error ("Failed to pretty print JSON: {}", e.getMessage (), e); } } return "No response body found!"; } This method accepts the String in the method parameter where the response object will be supplied. A check is performed using the if() condition to verify that the text supplied is not blank, null and it is not empty. If the condition is satisfied, then the ObjectMapper class from the Jackson Databind dependency is instantiated. Next, the text value of the response is read, and it is converted and returned as the JSON pretty print format using the writerWithDefaultPrettyPrinter() and writeValueAsString() methods of the ObjectMapper class. If the response is null, empty, and blank, it will print the message “No response body found!” and the method will be exited. How to Use the Logger in the API Automation Tests The Logger class needs to be instantiated and its respective methods need to be called in order to get the response details printed while the tests are executed. We need to make sure that we don’t write duplicate code everywhere in the tests to get the response details logged. In order to handle this, we would be using the BaseTest class and creating a new method, logResponse(APIResponse response). This method will accept the APIResponse as parameter, and the logResponseDetails() method will be called after instantiating the Logger class. Java public class BaseTest { //... protected void logResponse (final APIResponse response) { final Logger logger = new Logger (response); logger.logResponseDetails (); } } As the BaseTest class is extended to all the Test classes; it becomes easier to call the methods directly in the test class. The HappyPathTests class that we have used in previous blogs for adding happy scenario tests for testing GET, POST, PUT, PATCH, and DELETE requests already extends the BaseTest class. Let’s print the response logs for the POST and GET API request test. The testShouldCreateNewOrders() verifies that the new orders are created successfully. Let’s add the logResponse() method to this test and get the response printed in the logs. Java public class HappyPathTests extends BaseTest{ @Test public void testShouldCreateNewOrders() { final int totalOrders = 4; for (int i = 0; i < totalOrders; i++) { this.orderList.add(getNewOrder()); } final APIResponse response = this.request.post("/addOrder", RequestOptions.create() .setData(this.orderList)); logResponse (response); //... // Assertion Statements... } } The logResponse() method will be called after the POST request is sent. This will enable us to know what response was received before we start performing assertions. The testShouldGetAllOrders() verifies the GET /getAllOrder API request. Let’s add the logResponse() method to this test and check the response logs getting printed. Java public class HappyPathTests extends BaseTest{ @Test public void testShouldGetAllOrders() { final APIResponse response = this.request.get("/getAllOrders"); logResponse (response); final JSONObject responseObject = new JSONObject(response.text()); final JSONArray ordersArray = responseObject.getJSONArray("orders"); assertEquals(response.status(), 200); assertEquals(responseObject.get("message"), "Orders fetched successfully!"); assertEquals(this.orderList.get(0).getUserId(), ordersArray.getJSONObject(0).get("user_id")); assertEquals(this.orderList.get(0).getProductId(), ordersArray.getJSONObject(0).get("product_id")); assertEquals(this.orderList.get(0).getTotalAmt(), ordersArray.getJSONObject(0).get("total_amt")); } } The logResponse() method is called after the GET request is sent and will print the response logs in the console. Test Execution The tests will be executed in order where POST request will be executed first so new orders are created and then the GET request will be executed. It will be done using the testng-restfulecommerce-postandgetorder.xml file. XML <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> <suite name="Restful ECommerce Test Suite"> <test name="Testing Happy Path Scenarios of Creating and Updating Orders"> <classes> <class name="io.github.mfaisalkhatri.api.restfulecommerce.HappyPathTests"> <methods> <include name="testShouldCreateNewOrders"/> <include name="testShouldGetAllOrders"/> </methods> </class> </classes> </test> </suite> On executing the above testng-restfulecommerce-postandgetorder.xml file, the POST, as well as GET API requests, are executed, and the response is printed in the console, which can be seen in the screenshots below. POST API Response Logs GET API Response Logs It can be seen from the screenshots that the response logs are printed correctly in the console and can now help us know the exact results of the test execution. Summary Adding a custom logger in the project can help in multiple ways. It will provide us with the details of the test data that was processed along with the final output, giving us control over the tests. It also helps in debugging the issue for failed tests and finding a fix quickly for it. If the response data is readily available we can quickly find the pattern of the issue and try for a quick fix. As Playwright does not provide any method for logging the response details, we can add our own custom logger that can help us in fetching the required details. Happy testing!

By Faisal Khatri CORE

Building applications for programmatically editing Open Office XML (OOXML) documents like PowerPoint, Excel, and Word has never been easier to accomplish. Depending on the scope of their projects, Java developers can leverage open-source libraries in their code — or plugin-simplified API services — to manipulate content stored and displayed in the OOXML structure. Introduction In this article, we’ll specifically discuss how PowerPoint Presentation XML (PPTX) files are structured, and we’ll learn the basic processes involved in navigating and manipulating PPTX content. We’ll transition into talking about a popular open-source Java library for programmatically manipulating PPTX files (specifically, replacing instances of a text string), and we’ll subsequently explore a free third-party API solution that can help simplify that process and reduce local memory consumption. How are PowerPoint PPTX Files Structured? Like all OOXML files, PowerPoint PPTX files are structured as ZIP archives containing a series of hierarchically organized XML files. They’re essentially a series of directories, most of which are responsible for storing and arranging the resources we see when we open presentations in the PowerPoint application (or any PPTX file reader). PPTX archives start with a basic root structure, where the various content types we see in a PowerPoint (e.g., multimedia content) are neatly defined. The heart of a PPTX document resides at the directory level, with components like slides (e.g., firstSlide.xml, secondSlide.xml, etc.), slide layouts (e.g., templates), slide masters (e.g., global styles and placeholders), and other content (e.g., charts, media, and themes) clearly organized. The relationships between interdependent components in a PPTX file are stored in .rels XML files within the _rels directory. These relationship files automatically update when changes are made to slides or other content. With this file structure in mind, let’s imagine we wanted to manually replace a string of text within a PowerPoint slide without opening the file in PowerPoint or any other PPTX reader. To do that, we would first convert the PPTX archive to a ZIP file (with a .zip extension) and unzip its contents. After that, we would check the ppt/presentation.xml file, which lists the slides in order, and we would then navigate to the ppt/slides/ directory to locate our target slide (e.g., secondSlide.xml). To modify the slide, we would open secondSlide.xml, locate the text run we needed (typically structured as <a:t> “string” </a:t> within an <a:r></a:r> tag), and replace the text content with a new string. We would then check the _rels directory to ensure the slide relationships remained intact; after that, we would repackage the file as a ZIP archive and reintroduce a .pptx extension. All done! Changing PPTX Files Programmatically in Java To handle the exact same process in Java, we would have to consider a few different possibilities depending on the context. Obviously, nobody wants to manually map the entire OOXML structure to a custom Java program on the fly — so we’d have to determine whether using an open-source library or a plug-and-play API service would make more sense based on our project constraints. If we chose the open-source route, Apache POI would be a great option. Apache POI is an open-source Java API designed specifically to help developers work with Microsoft documents, including PowerPoint PPTX (and also Excel XLSX, Word DOCX, etc.). For a project concerned with PPTX files, we would first import relevant Apache POI classes for a PowerPoint project (e.g., XMLSlideShow, XSLFSlide, and XSLFTextShape). We would then load the PPTX file using the XMLSlideShow class, invoke the getSlides() method, filter text content with the XSLFTextShape class, and invoke the getText() and setText() methods to replace a particular string. This would work just fine, but it's worth noting that the challenge with using an open-source library like Apache POI is the way memory is handled. Apache POI loads all data into local memory, and although there are some workarounds — e.g., increasing JVM heap size or implementing stream-based APIs — we’re likely consuming a ton of resources dealing with large PPTX files at scale. Leveraging a Third-Party API Solution If we can’t handle a PPTX editing workflow locally, we might benefit from a cloud-based API solution. This type of solution offloads the bulk of our file processing to an external server and returns the result, reducing overhead. As a side benefit, it also simplifies the process of structuring our string replacement request. We’ll look at one API solution below. The below ready-to-run example Java code can be used to call a free web API that replaces all instances of a string found in a PPTX document. The API is free to use with a free API key, and the parameters are extremely straightforward to work with. To structure our API call, we’ll begin by incorporating the client library in our Maven project. We’ll add the following (JitPack) repository reference to our pom.xml: XML <repositories> <repository> <id>jitpack.io</id> <url>https://jitpack.io</url> </repository> </repositories> Next, we’ll add the below dependency reference to our pom.xml: XML <dependencies> <dependency> <groupId>com.github.Cloudmersive</groupId> <artifactId>Cloudmersive.APIClient.Java</artifactId> <version>v4.25</version> </dependency> </dependencies> With that out of the way, we’ll now copy the below import classes and add them to the top of our file: Java // Import classes: //import com.cloudmersive.client.invoker.ApiClient; //import com.cloudmersive.client.invoker.ApiException; //import com.cloudmersive.client.invoker.Configuration; //import com.cloudmersive.client.invoker.auth.*; //import com.cloudmersive.client.EditDocumentApi; Now, we’ll use the below code to initialize the API client and subsequently configure API key authorization. The setAPIKey()method will capture our API key string: Java ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: Apikey ApiKeyAuth Apikey = (ApiKeyAuth) defaultClient.getAuthentication("Apikey"); Apikey.setApiKey("YOUR API KEY"); // Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null) //Apikey.setApiKeyPrefix("Token"); Finally, we’ll use the code below to instantiate the API client, configure the replacement operation, execute the replacement process (returning a byte[] array), and catch/log errors: Java EditDocumentApi apiInstance = new EditDocumentApi(); ReplaceStringRequest reqConfig = new ReplaceStringRequest(); // ReplaceStringRequest | Replacement document configuration input try { byte[] result = apiInstance.editDocumentPptxReplace(reqConfig); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling EditDocumentApi#editDocumentPptxReplace"); e.printStackTrace(); } The JSON below defines the structure of our request; we’ll use this in our code to configure the parameters of our string replacement operation. JSON { "InputFileBytes": "string", "InputFileUrl": "string", "MatchString": "string", "ReplaceString": "string", "MatchCase": true } We can prepare a PPTX document for this API request by reading the file into a byte array and converting it to a Base64-encoded string. Conclusion In this article, we discussed the way PowerPoint PPTX files are structured and how that structure lends itself to straightforward PowerPoint document editing outside of a PPTX reader. We then suggested the Apache POI library as an open-source solution for Java developers to programmatically replace strings in PPTX files, before also exploring a free third-party API solution for handling the same process at less local memory cost. As a quick final note — for anyone interested in similar articles focused on Excel XLSX or Word DOCX documents, I’ve covered those topics in prior articles over the years.

By Brian O'Neill CORE

Encountering connection problems while accessing a MySQL server is a common challenge for database users. These issues often arise due to incorrect configuration, user permissions, or compatibility problems. Below are the most common errors and their solutions to help you resolve connection issues efficiently. 1. Error: Host ‘xxx.xx.xxx.xxx’ is not allowed to connect to this MySQL server Cause This error indicates that the MySQL server does not permit the specified host or user to access the database. It is typically due to insufficient privileges assigned to the user or client host. Solution To resolve this issue, grant the required privileges to the user from the MySQL command line: MySQL mysql> USE mysql; mysql> GRANT ALL ON *.* TO 'urUser'@'[urhostname]' IDENTIFIED BY 'urpassword'; mysql> FLUSH PRIVILEGES; Replace urUser and urpassword with your actual username and password.Replace [urhostname] with the hostname or IP address of the client trying to connect. If the problem persists, verify that networking is enabled on the MySQL server. For newer MySQL versions, use the MySQL Server Instance Configuration Tool to enable TCP/IP networking. Ensure the TCP/IP networking option is checked.Specify the port (default is 3306) and create a firewall exception for this port. 2. Error: Unable to connect to any of the specified hosts Cause This generic error can occur for several reasons, including server misconfiguration or incorrect network settings. Solution Try the following steps to resolve the issue: 1. Verify MySQL Server is Running On Windows, ensure the MySQL Service is running.On Linux, check the server status with: MySQL systemctl status mysql 2. Enable TCP/IP Networking Open the MySQL configuration file (typically named my.ini or my.cnf).Ensure the line skip-networking is commented out or removed.Restart the MySQL server after making changes. 3. Check the Port Number MySQL servers usually run on port 3306.Verify the port number in the MySQL configuration file (my.ini or my.cnf). 4. Firewall Rules Ensure your firewall is not blocking MySQL’s port. Add an exception for port 3306 if needed. 3. Error: Access denied for user ‘UserName’@’HostName’ (using password: YES) Cause This error is generally caused by incorrect login credentials, such as a mistyped username or password. Solution Double-check the username and password you’re using. Ensure that: The username and password match those set in MySQL.The user has been granted access to the specific database or host. MySQL GRANT ALL ON dbName.* TO 'UserName'@'HostName' IDENTIFIED BY 'password'; FLUSH PRIVILEGES; Replace dbName, UserName, HostName, and password with your actual database name, username, hostname, and password. 4. Error: Client does not support authentication protocol requested by server; consider upgrading MySQL client Cause This error is common when connecting to MySQL 8.0, as it uses a new authentication protocol called caching_sha2_password. Older MySQL clients or tools may not support this protocol. Solution Option 1: Upgrade Your Client Upgrade to the latest version of your MySQL client or tool. For example, if you’re using Data Loader, upgrade to version 4.9 or later, which supports the newer authentication protocol. Option 2: Workaround for Older Clients If you cannot upgrade your client, follow these steps to create a compatible user in MySQL 8.0: 1. Create a new user. MySQL mysql> CREATE USER 'user1'@'localhost' IDENTIFIED BY 'passxxx'; 2. Grant required privileges. MySQL mysql> GRANT ALL ON *.* TO 'user1'@'localhost' IDENTIFIED BY 'passxxx'; 3. Change the authentication method. MySQL mysql> ALTER USER 'user1'@'localhost' IDENTIFIED WITH mysql_native_password BY 'passxxx'; This command reverts the user’s authentication method to the legacy mysql_native_password protocol, which is compatible with older clients. You can now use this user (user1) to connect to MySQL 8.0 with older tools or libraries. Troubleshooting MySQL connection issues often involves addressing configuration settings, user permissions, or compatibility between the client and server. By following the solutions outlined above, you can resolve common errors such as permission denials, misconfiguration of hosts, or protocol mismatches. For ongoing database management, ensure your MySQL server and tools are kept up to date, and review user privileges and network settings periodically to avoid future connection problems.

Chrome extensions have traditionally been built using JavaScript, HTML, and CSS. However, with the rise of WebAssembly (Wasm), we can now leverage Rust's performance, safety, and modern development features in browser extensions. In this tutorial, we will create a simple Chrome extension that uses Rust compiled to WebAssembly. Prerequisites Before we begin, you'll need: Plain Text # Install Rust and Cargo curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Install Node.js and npm (using nvm for version management) curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash && nvm install 23 # Install wasm-pack (required for WebAssembly compilation and JavaScript bindings) cargo install wasm-pack Note: While Rust and Cargo provide basic WebAssembly support, we specifically need wasm-pack for this project as it handles the WebAssembly compilation process and generates optimized JavaScript bindings for browser integration. Project Structure Our Chrome extension will have the following structure: Plain Text rust-chrome-extension/ ├── Cargo.toml ├── manifest.json ├── package.json ├── popup.html ├── popup.js └── src/ └── lib.rs Setting Up the Project 1. First, create a new Rust project: Plain Text cargo new rust-chrome-extension --lib cd rust-chrome-extension 2. Update your Cargo.toml with the necessary dependencies: Plain Text [package] name = "rust-chrome-extension" version = "0.1.0" edition = "2021" [lib] crate-type = ["cdylib"] [dependencies] wasm-bindgen = "0.2" 3. Create a simple Rust function in src/lib.rs: Plain Text use wasm_bindgen::prelude::*; #[wasm_bindgen] pub fn greet(name: &str) -> String { format!("Hello, {}! From Rust!", name) } 4. Create manifest.json for the Chrome extension: Plain Text { "manifest_version": 3, "name": "Rust Chrome Extension", "version": "1.0", "description": "A simple Chrome extension using Rust and WebAssembly", "action": { "default_popup": "popup.html" } } 5. Create popup.html: Plain Text <!DOCTYPE html> <html> <head> <title>Rust Chrome Extension</title> </head> <body> <input type="text" id="name" placeholder="Enter your name"> <button id="greet">Greet</button> <div id="output"></div> <script src="pkg/rust_chrome_extension.js"></script> <script src="popup.js"></script> </body> </html> 6. Create popup.js: Plain Text import init, { greet } from './pkg/rust_chrome_extension.js'; async function main() { await init(); document.getElementById('greet').addEventListener('click', () => { const name = document.getElementById('name').value; const output = greet(name); document.getElementById('output').textContent = output; }); } main(); Building the Extension 1. Build the Rust code to WebAssembly: Plain Text wasm-pack build --target web 2. Load the extension in Chrome: Open Chrome and navigate to chrome://extensions/Enable "Developer mode"Click "Load unpacked" and select your project directory How It Works Let's break down the key components: 1. Rust Code (lib.rs) We use wasm-bindgen to create JavaScript bindings for our Rust functionThe #[wasm_bindgen] attribute makes our function available to JavaScriptOur simple greet function takes a name parameter and returns a formatted string 2. HTML (popup.html) Creates a basic UI with an input field and buttonLoads the WebAssembly module and JavaScript code 3. JavaScript (popup.js) Initializes the WebAssembly moduleSets up event listeners for user interactionCalls our Rust function and displays the result Testing the Extension After loading the extension in Chrome: Click the extension icon to open the popupEnter a name in the input fieldClick the "Greet" buttonYou should see a greeting message generated by the Rust code! Demo Here's what our extension looks like in action: Sample output when you enter "Vivek" and click the Greet button: Hello, Vivek! From Rust! For a live demonstration and more examples, check out the demo folder in the GitHub repository. Why Choose Rust for Chrome Extensions? Building browser extensions with this system programming language offers compelling advantages that set it apart from traditional JavaScript-based development. Here's a detailed breakdown of key benefits: FeatureDescriptionExamplePerformanceNative-speed execution through WebAssembly (WASM) compilationProcessing large datasets or images runs significantly faster than JavaScript, delivering superior user experienceMemory SafetyAdvanced ownership model eliminates common bugs and vulnerabilitiesPrevents null pointer dereferences and memory leaks that typically crash JavaScript extensionsConcurrencyBuilt-in support for safe multi-threadingFetch data from multiple APIs simultaneously without race conditionsCross-Browser CompatibilityWASM compilation ensures consistent performance across browsersYour extension works seamlessly on Chrome, Firefox, and EdgeFramework IntegrationSeamless handling of complex computations alongside modern front-end frameworksUse React for UI while performing intensive calculations in the background Additional benefits include: Modern tooling: Access to a powerful ecosystem and efficient package management through cargoType system: Catch errors at compile time rather than runtimeGrowing ecosystem: Expanding collection of libraries specifically for browser extension developmentFuture-proof: As WebAssembly evolves, this technology stack becomes increasingly valuable for web development Complete Code The complete source code for this tutorial is available on rust-chrome-extension. Conclusion We've successfully created a simple Chrome extension that leverages Rust and WebAssembly. While this example is basic, it demonstrates how to combine these technologies and sets the foundation for more complex extensions. The combination of Rust's safety and performance with Chrome's extension platform opens up new possibilities for building powerful browser extensions. As WebAssembly continues to evolve, we can expect to see more developers adopting this approach. Give it a try in your next Chrome extension project — you might be surprised by how easy it is to get started with Rust and WebAssembly!

Google Cloud Analytics Hub is a tool built on BigQuery that enables seamless data sharing across the organization by making it easier to share and access datasets. Analytics Hub makes it easy to discover public, private, and internally shared data sources. Accessing Public Datasets in Analytics Hub Navigate to the Google Cloud console using the URL "https://console.cloud.google.com," search for BigQuery, and select BigQuery. In the BigQuery console, click on Analytics Hub, and click on Search Listings. Search for trees to search for listings named "trees" and click on Street Trees Listing. Click on Subscribe to subscribe to the listing. Select the desired project for your linked dataset by clicking on Browse. Once you've made your selection, provide the linked dataset name and click Save to continue. Click on Go to Linked Dataset. Expand the street_trees dataset in BigQuery, select the street_trees table, click on Query, and click on In new tab. Add “*” to the Query and click Run. It's that simple to query linked datasets in BigQuery. Creating Dataset in BigQuery In BigQuery, datasets provide a logical structure for managing tables and views within a project. Before loading data into BigQuery, users must create at least one dataset to store their tables or views. Navigate to BigQuery Studio and click on Create Dataset to create dataset in BigQuery. Specify the Dataset ID as dzone-dataset, then click on Create Dataset. Creating a Table in BigQuery BigQuery organizes data in Tables where data is stored in rows and columns. In BigQuery Studio, run the Query below by clicking on Run. Plain Text CREATE OR REPLACE TABLE `dzone_dataset.analytics_hub_table` AS SELECT * FROM `dzone-tutorial.street_trees.street_trees` LIMIT 1000 Notice that the new table 'analytics_hub_table' is created. Creating Analytics Hub Data Exchange Data exchanges streamline data sharing by providing a structured environment for publishing and accessing data. Data exchange is a catalog of available datasets. Analytics Hub allows publishers and administrators to manage subscriber access at both the exchange and the listing levels. The Analytics Hub subscriber can browse data exchanges, discover accessible data, and subscribe to the shared resources. This method eliminates the requirement to explicitly grant access to the underlying shared resources. When creating a data exchange, a primary contact email can be assigned, providing a means for users to contact the owner with questions or concerns about the data exchange. Click on Create Exchange to create an exchange in Analytics Hub. Specify the Display Name as the name you want to specify for the exchange. Users can click on the Toggle to make the exchange publicly discoverable. Click on Create Exchange. Users can specify the administrators for exchange who can manage the listings, specify the publishers who can publish and manage the listings, specify the subscribers who can subscribe to the listings, or specify viewers who can view listings and exchange permissions. We can click on Skip. Notice the dzone-tutorial-exchange has been created. Click on the dzone-tutorial-exchange. Listing is a dataset that we want to share with Analytics Hub and a listing could be a public listing or private listing. Click on Create Listing. Select the Resource Type as BigQuery Dataset, select the dataset we have created earlier by typing the dataset name, in our example we will select dzone-tutorial.dzone_dataset. Click on Next. Provide the Display Name as Analytics Hub Sample Trees Data. In the markdown provide the code "# Sample Dataset For Trees Data" and click on Next. In Analytics Hub, the Publisher should have an Analytics Hub Listing Admin role or Analytics Hub Publisher role to create the listing. We can provide the listing contact information, such as primary contact, provider name, and publisher name, by clicking on Next. In this example, we are leaving it empty. Click on Publish. Click on Search Listings to search for our published listing. Search for dzone and click on the Analytics Hub Sample Trees Data tile. To discover and subscribe to the listings the user should have the Analytics Hub subscriber role. If you are logged into Analytics Hub as a user that has Analytics Hub subscriber then we can click on Subscribe. Specify the linked dataset name as sample_analytics_hub_sample_trees_data and click on Save. Click on Go to Linked Dataset. Click on the analytics_hub_table and click on Query. Add * to the Query and click on Run. Notice that we are able to see the linked dataset. This is how data is published and subscribed to Analytics Hub. Summary Google Analytics Hub provides a secure and seamless way to share and access datasets, both public and private. Built on BigQuery, Google Analytics Hub simplifies data sharing by allowing users to easily create and subscribe to listings, making data readily available for analysis within their projects.