Testing, Deployment, and Maintenance

Writing software is an act of creation, and Android development is no exception. It’s about more than just making something work. It’s about designing applications that can grow, adapt, and remain manageable over time. As an Android developer who has faced countless architectural challenges, I’ve discovered that adhering to the SOLID principles can transform even the most tangled codebases into clean systems. These are not abstract principles, but result-oriented and reproducible ways to write robust, scalable, and maintainable code. This article will provide insight into how SOLID principles can be applied to Android development through real-world examples, practical techniques, and experience from the Meta WhatsApp team. Understanding SOLID Principles The SOLID principles, proposed by Robert C. Martin, are five design principles for object-oriented programming that guarantee clean and efficient software architecture. Single Responsibility Principle (SRP). A class should have one and only one reason to change.Open/Closed Principle (OCP). Software entities should be open for extension but closed for modification.Liskov Substitution Principle (LSP). Subtypes must be substitutable for their base types.Interface Segregation Principle (ISP). Interfaces should be client-specific and not force the implementation of unused methods.Dependency Inversion Principle (DIP). High-level modules should depend on abstractions, not on low-level modules. Integrating these principles into Android development will allow us to create applications that are easier to scale, test, and maintain. Single Responsibility Principle (SRP): Streamlining Responsibilities The Single Responsibility Principle is the foundation of writing maintainable code. It states that each class must have a single concern it takes responsibility for. A common anti-pattern is considering activities or fragments to be some "God classes" that handle responsibilities starting from UI rendering, then data fetching, error handling, etc. This approach makes a test and maintenance nightmare. With the SRP, separate different concerns into different components: for example, in an app for news, create or read news. Java class NewsRepository { fun fetchNews(): List<News> { // Handles data fetching logic } } class NewsViewModel(private val newsRepository: NewsRepository) { fun loadNews(): LiveData<List<News>> { // Manages UI state and data flow } } class NewsActivity : AppCompatActivity() { // Handles only UI rendering } Every class has only one responsibility; hence, it’s easy to test and modify without having side effects. In modern Android development, SRP is mostly implemented along with the recommended architecture using Jetpack. For example, logic related to data manipulation logic might reside inside ViewModel, while the Activities or Fragments should just care about the UI and interactions. Data fetching might be delegated to some separate Repository, either from local databases like Room or network layers such as Retrofit. This reduces the risk of UI classes bloat, since each component gets only one responsibility. Simultaneously, your code will be much easier to test and support. Open/Closed Principle (OCP): Designing for Extension The Open/Closed Principle declares that a class should be opened for extension but not for modification. It is more reasonable for Android applications since they constantly upgrade and add new features. The best examples of how to use the OCP principle in Android applications are interfaces and abstract classes. For example: Java interface PaymentMethod { fun processPayment(amount: Double) } class CreditCardPayment : PaymentMethod { override fun processPayment(amount: Double) { // Implementation for credit card payments } } class PayPalPayment : PaymentMethod { override fun processPayment(amount: Double) { // Implementation for PayPal payments } } Adding new payment methods does not require changes to existing classes; it requires creating new classes. This is where the system becomes flexible and can be scaled. In applications created for Android devices, the Open/Closed Principle is pretty useful when it comes to feature toggles and configurations taken dynamically. For example, in case your app has an AnalyticsTracker base interface that reports events to different analytics services, Firebase and Mixpanel, and custom internal trackers, every new service can be added as a separate class without changes to the existing code. This keeps your analytics module open for extension — you can add new trackers — but closed for modification: you don’t rewrite existing classes every time you add a new service. Liskov Substitution Principle (LSP): Ensuring Interchangeability The Liskov Substitution Principle states that subclasses should be substitutable for their base classes, and the application’s behavior must not change. In Android, this principle is fundamental to designing reusable and predictable components. For example, a drawing app: Java abstract class Shape { abstract fun calculateArea(): Double } class Rectangle(private val width: Double, private val height: Double) : Shape() { override fun calculateArea() = width * height } class Circle(private val radius: Double) : Shape() { override fun calculateArea() = Math.PI * radius * radius } Both Rectangle and Circle can be replaced by any other one interchangeably without system failure, which means that the system is flexible and follows LSP. Consider Android’s RecyclerView.Adapter subclasses. Each subclass of the adapter extends from RecyclerView.Adapter<VH> and overrides core functions like onCreateViewHolder, onBindViewHolder, and getItemCount. The RecyclerView can use any subclass interchangeably as long as those methods are implemented correctly and do not break the functionality of your app. Here, the LSP is maintained, and your RecyclerView can be flexible to substitute any adapter subclass at will. Interface Segregation Principle (ISP): Lean and Focused Interfaces In larger applications, it is common to define interfaces with too much responsibility, especially around networking or data storage. Instead, break them into smaller, more targeted interfaces. For example, an ApiAuth interface responsible for user authentication endpoints should be different from an ApiPosts interface responsible for blog posts or social feed endpoints. This separation will prevent clients that need only the post-related methods from being forced to depend on and implement authentication calls, hence keeping your code and test coverage leaner. The Interface Segregation Principle means that instead of having big interfaces, several smaller, focused ones should be used. The principle prevents situations where classes implement unnecessary methods. For example, rather than having one big interface representing users’ actions, consider Kotlin code: Kotlin interface Authentication { fun login() fun logout() } interface ProfileManagement { fun updateProfile() fun deleteAccount() } Classes that implement these interfaces can focus only on the functionality they require, thus cleaning up the code and making it more maintainable. Dependency Inversion Principle (DIP): Abstracting Dependencies The Dependency Inversion Principle promotes decoupling by ensuring high-level modules depend on abstractions rather than concrete implementations. This principle perfectly aligns with Android’s modern development practices, especially with dependency injection frameworks like Dagger and Hilt. For example: Kotlin class UserRepository @Inject constructor(private val apiService: ApiService) { fun fetchUserData() { // Fetches user data from an abstraction } } Here, UserRepository depends on the abstraction ApiService, making it flexible and testable. This approach allows us to replace the implementation, such as using a mock service during testing. Frameworks such as Hilt, Dagger, and Koin facilitate dependency injection by providing a way to supply dependencies to Android components, eliminating the need to instantiate them directly. In a repository, for instance, instead of instantiating a Retrofit implementation, you will inject an abstraction, for example, an ApiService interface. That way, you could easily switch the network implementation, for instance, an in-memory mock service for local testing, and would not need to change anything in your repository code. In real-life applications, you can find that classes are annotated with @Inject or @Provides to provide these abstractions, hence making your app modular and test-friendly. Practical Benefits of SOLID Principles Adopting SOLID principles in Android development yields tangible benefits: Improved testability. Focused classes and interfaces make it easier to write unit tests.Enhanced maintainability. Clear separation of concerns simplifies debugging and updates.Scalability. Modular designs enable seamless feature additions.Collaboration. Well-structured code facilitates teamwork and reduces onboarding time for new developers.Performance optimization. Lean, efficient architectures minimize unnecessary processing and memory usage. Real-World Applications In feature-rich applications, such as e-commerce or social networking apps, the application of the SOLID principles can greatly reduce the risk of regressions every time a new feature or service is added. For example, if a new requirement requires an in-app purchase flow, you can introduce a separate module to implement the required interfaces (Payment, Analytics) without touching the existing modules. This kind of modular approach, driven by SOLID, allows your Android app to quickly adapt to market demands and keeps the codebase from turning into spaghetti over time. While working on a large project that requires many developers to collaborate, it is highly recommended to keep a complex codebase with SOLID principles. For example, separating data fetching, business logic, and UI handling in the chat module helped reduce the chance of regressions while scaling the code with new features. Likewise, the application of DIP was crucial to abstract network operations, hence being able to change with almost no disruption between network clients. Conclusion More than a theoretical guide, the principles of SOLID are actually the practical philosophy for creating resilient, adaptable, and maintainable software. In Android development, with requirements changing nearly as often as technologies are, embracing these principles will allow you to write better code and build applications that are a joy to develop, scale, and maintain.

If there’s one thing Web3 devs can agree on, it’s that Sybils suck. Bots and fake accounts are ruining airdrops, gaming economies, DAOs, and DeFi incentives. Everyone’s trying to fight them, but the solutions are either too centralized and non-private (KYC) or too easy to game (staking-based anti-Sybil tricks). That’s where Biomapper comes in handy — an on-chain tool that links one EVM account to one human for verifying that users are real, unique humans without KYC or exposing personal data. How Biomapper Works (Without Screwing Up Privacy) Alright, so Biomapper is cross-chain — but how does it actually work? More importantly, how does it verify that someone is a real human without exposing their real-world identity? Here’s the TL;DR: User scans their face using the Biomapper App.Their biometric data is encrypted inside a Confidential Virtual Machine (CVM) (which means no one — not even Humanode — can see or access it).A Bio-token is generated and linked to their EVM wallet address.They bridge their bio-token to the required chain.When they interact with a dApp, the smart contract checks the Bio-token to confirm they’re a unique person.Done. No identity leaks, no personal data floating around — just proof that they’re not a bot. Why This Is Different from Other Anti-Sybil Methods No KYC. No passports, IDs, or personal data needed.No staking requirements. You can’t just buy your way past the system.No centralized verification authority. No one controls the user list.Privacy-first. Biometrics are never stored or shared with dApps. How Projects on Avalanche Can Use Biomapper Once a user is biomapped, any EVM dApp can check their uniqueness with a single smart contract call. That means: Airdrop contracts. Only real humans can claim.DAO voting. One person, one vote. No governance takeovers.Game reward systems. No multi-account farming.NFT whitelists. Verified users only, without KYC.And many more use cases. It’s Sybil resistance without the usual headaches. And integrating it into your dApp? That’s easy. Let’s go step-by-step on how to set it up. How to Integrate Your dApps With Biomapper Smart Contracts Getting Started Before you begin, make sure you're familiar with the core Biomapper concepts, such as: Generations. CVMs with Biomapping data resets periodically.General Integration Flow. Users biomap once, bridge it to the specific chain, and can be verified across dApps. What You Need to Do Write a smart contract that interacts with Bridged Biomapper on Avalanche C-Chain.Add a link to the Biomapper UI on your frontend, so users can complete their biomapping. Installation: Set Up Your Development Environment Before you begin, install the required Biomapper SDK dependencies. Install the Biomapper SDK Humanode Biomapper SDK provides interfaces and useful utilities for developing smart contracts and dApps that interact with the Humanode Biomapper and Bridged Biomapper smart contracts. It is usable with any tooling the modern EVM smart contract development ecosystem provides, including Truffle, Hardhat, and Forge. It is open-source and is available on GitHub, you will find the links to the repo, examples, and the generated code documentation below. Using npm (Hardhat/Node.js projects): YAML npm install --save @biomapper-sdk/core @biomapper-sdk/libraries @biomapper-sdk/events Using yarn: YAML yarn add @biomapper-sdk/core @biomapper-sdk/libraries @biomapper-sdk/events Using Foundry: If you're using Foundry, add the Biomapper SDK as a dependency: YAML forge install humanode-network/biomapper-sdk These packages allow you to interact with Biomapper smart contracts and APIs. Smart Contract Development The next step is to integrate Bridged Biomapper into your smart contract. Step 1: Import Biomapper Interfaces and Libraries In your Solidity smart contract, import the necessary interfaces and libraries from the Biomapper SDK. Rust // Import the IBridgedBiomapperRead interface import { IBridgedBiomapperRead } from "@biomapper-sdk/core/IBridgedBiomapperRead.sol"; // Import the IBiomapperLogRead interface import { IBiomapperLogRead } from "@biomapper-sdk/core/IBiomapperLogRead.sol"; // Import the BiomapperLogLib library import { BiomapperLogLib } from "@biomapper-sdk/libraries/BiomapperLogLib.sol"; These imports provide your smart contract with the necessary functions to verify user uniqueness and access biomapping logs. Step 2: Use Bridged Biomapper on Avalanche Since Humanode has already deployed the Bridged Biomapper contract on Avalanche C-Chain, your dApp should interact with it instead of deploying a new Biomapper contract. Smart contract example: Rust pragma solidity ^0.8.0; // Import the Bridged Biomapper Read interface import "@biomapper-sdk/core/IBridgedBiomapperRead.sol"; contract MyDapp { IBridgedBiomapperRead public biomapper; constructor(address _biomapperAddress) { biomapper = IBridgedBiomapperRead(_biomapperAddress); } function isUserUnique(address user) public view returns (bool) { return biomapper.isBridgedUnique(user); } } What this does: Connects your contract to the official Bridged Biomapper contract on Avalanche.Allows your contract to verify if a user has been biomapped and is unique. To access the contract addresses, APIs, and more information about the particular contracts, functions, and events from the official Biomapper SDK documentation. Step 3: Using Mock Contracts for Local Development For local testing, you can use the MockBridgedBiomapper contract. This allows developers to simulate the integration before deploying to testnet or mainnet. Example usage: Rust function generationsBridgingTxPointsListItem(uint256 ptr) external view returns (GenerationBridgingTxPoint memory); Refer to the Biomapper SDK Docs for technical details on using mock contracts. Step 4: Calling Biomapper Functions in Your Smart Contracts Checking user uniqueness: Before allowing a user to claim rewards or access features, verify whether they have a valid biomapping. Rust function isUnique(IBiomapperLogRead biomapperLog, address who) external view returns (bool); If the function returns false, prompt the user to complete the verification process. Implementing Unique User Verification Here’s an example smart contract that ensures each user is unique before accessing in-game rewards: Rust using BiomapperLogLib for IBiomapperLogRead; IBiomapperLogRead public immutable BIOMAPPER_LOG; IBridgedBiomapperRead public immutable BRIDGED_BIOMAPPER; mapping(address => bool) public hasClaimedReward; event RewardClaimed(address player); constructor(address biomapperLogAddress, address bridgedBiomapperAddress) { BIOMAPPER_LOG = IBiomapperLogRead(biomapperLogAddress); BRIDGED_BIOMAPPER = IBridgedBiomapperRead(bridgedBiomapperAddress); } function claimGameReward() public { require(!hasClaimedReward[msg.sender], "Reward already claimed"); require(BIOMAPPER_LOG.biomappingsHead(msg.sender) != 0, "User is not biomapped"); require(BRIDGED_BIOMAPPER.biomappingsHead(msg.sender) != 0, "User is not biomapped on bridged chain"); hasClaimedReward[msg.sender] = true; emit RewardClaimed(msg.sender); } } Frontend Integration Integrating Biomapper on the frontend is simple — just add a link to the Biomapper App so users can verify themselves. HTML <a href="https://biomapper.humanode.io" target="_blank"> Verify Your Uniqueness with Biomapper </a> What this does: Redirects users to the Biomapper App, where they scan their biometrics.Once verified, their wallet is biomapped and linked to their EVM address. Testing and Rollout Once you have verified your contract works locally, deploy it to Avalanche C-Chain. Summing Up With Humanode Biomapper live on Avalanche C-Chain, developers now have a privacy-preserving, Sybil-resistant way to verify real users without KYC. Whether for airdrops, DAO governance, gaming, or DeFi, Biomapper ensures fairness by preventing bots and multi-wallet exploits. Once integrated, your dApp is now protected against Sybil attacks while maintaining user privacy. To take it further: Get your dApp listed in the Biomapper App by reaching out to HumanodeDeploy on other EVM-compatible chains beyond AvalancheExplore Biomapper's cross-chain capability A more human Web3 is now possible. Start integrating today. For more details, visit the Biomapper SDK Docs and Biomapper docs.

There are scenarios where we would not want to use commercial large language models (LLMs) because the queries and data would go into the public domain. There are ways to run open-source LLMs locally. This article explores the option of running Ollama locally interfaced with the Sprint boot application using the SpringAI package. We will create an API endpoint that will generate unit test cases for the Java code that has been passed as part of the prompt using AI with the help of Ollama LLM. Running Open-Source LLM Locally 1. The first step is to install Ollama; we can go to ollama.com, download the equivalent OS version, and install it. The installation steps are standard, and there is nothing complicated. 2. Please pull the llama3.2 model version using the following: PowerShell ollama pull llama3.2 For this article, we are using the llama3.2 version, but with Ollama, we can run a number of other open-source LLM models; you can find the list over here. 3. After installation, we can verify that Ollama is running by going to this URL: http://localhost:11434/. You will see the following status "Ollama is running": 4. We can also use test containers to run Ollama as a docker container and install the container using the following command: PowerShell docker run -d -v ollama:/root/.ollama -p 11438:11438 --name ollama ollama/ollama Since I have Ollama, I have been running locally using local installation using port 11434; I have swapped the port to 11438. Once the container is installed, you can run the container and verify that Ollama is running on port 11438. We can also verify the running container using the docker desktop as below: SpringBoot Application 1. We will now create a SpringBoot application using the Spring Initializer and then install the SpringAI package. Please ensure you have the following POM configuration: XML <repositories> <repository> <id>spring-milestones</id> <name>Spring Milestones</name> <url>https://repo.spring.io/milestone</url> <snapshots> <enabled>false</enabled> </snapshots> </repository> <repository> <id>spring-snapshots</id> <name>Spring Snapshots</name> <url>https://repo.spring.io/snapshot</url> <releases> <enabled>false</enabled> </releases> </repository> </repositories> <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-ollama-spring-boot-starter</artifactId> <version>1.0.0-SNAPSHOT</version> <type>pom</type> <scope>import</scope> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.springframework.ai</groupId> <artifactId>spring-ai-ollama</artifactId> <version>1.0.0-M6</version> </dependency> </dependencies> 2. We will then configure the application.properties for configuring the Ollama model as below: Properties files spring.application.name=ollama spring.ai.ollama.base-url=http://localhost:11434 spring.ai.ollama.chat.options.model=llama3.2 3. Once the spring boot application is running, we will write code to generate unit tests for the Java code that is passed as part of the prompt for the spring boot application API. 4. We will first write a service that will interact with the Ollama model; below is the code snippet: Java @Service public class OllamaChatService { @Qualifier("ollamaChatModel") private final OllamaChatModel ollamaChatModel; private static final String INSTRUCTION_FOR_SYSTEM_PROMPT = """ We will using you as a agent to generate unit tests for the code that is been passed to you, the code would be primarily in Java. You will generate the unit test code and return in back. Please follow the strict guidelines If the code is in Java then only generate the unit tests and return back, else return 'Language not supported answer' If the prompt has any thing else than the Java code provide the answer 'Incorrect input' """; public OllamaChatService(OllamaChatModel ollamaChatClient) { this.ollamaChatModel = ollamaChatClient; } public String generateUnitTest(String message){ String responseMessage = null; SystemMessage systemMessage = new SystemMessage(INSTRUCTION_FOR_SYSTEM_PROMPT); UserMessage userMessage = new UserMessage(message); List<Message> messageList = new ArrayList<>(); messageList.add(systemMessage); messageList.add(userMessage); Prompt userPrompt = new Prompt(messageList); ChatResponse extChatResponse = ollamaChatModel.call(userPrompt); if (extChatResponse != null && extChatResponse.getResult() != null && extChatResponse.getResult().getOutput() != null){ AssistantMessage assistantMessage = ollamaChatModel.call(userPrompt).getResult().getOutput(); responseMessage = assistantMessage.getText(); } return responseMessage; } } 5. Please take a look at the PROMPT_INSTRUCTIONS; we'd like to define the chat agent's purpose and responsibility. We are enforcing the responsibility to generate unit test code for Java. If anything else is sent, the Prompt answer will be returned as "Incorrect Input." 6. Then, we will build an API endpoint, which will interact with the chat service. Java @RestController @RequestMapping("/api/ai/ollama") public class OllamaChatController { @Autowired OllamaChatService ollamaChatService; @PostMapping("/unit-test") public ChatResponse generateUnitTests(@RequestBody ChatRequest request) { String response = this.ollamaChatService.generateUnitTest(request.getPrompt()); ChatResponse chatResponse = new ChatResponse(); chatResponse.setMessage(response); return chatResponse; } } Running the API Endpoint Generate unit tests. Below is the sample output from the API for generating unit tests for the random number method. Here are the unit tests for the given Java code: Java import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*; import java.util.Random; public class RandomNumberGeneratorTest { @Test public void testGetRandomNumberMinAndMaxSame() { int min = 10; int max = 10; try { int randomNumber = getRandomNumber(min, max); fail("Expected IllegalArgumentException"); } catch (IllegalArgumentException e) { assertEquals("Max must be greater than min", e.getMessage()); } } @Test public void testGetRandomNumberMinLessThanZero() { int min = -10; int max = 10; try { int randomNumber = getRandomNumber(min, max); fail("Expected IllegalArgumentException"); } catch (IllegalArgumentException e) { assertEquals("Max must be greater than min", e.getMessage()); } } @Test public void testGetRandomNumberMinGreaterThanMax() { int min = 10; int max = 5; try { int randomNumber = getRandomNumber(min, max); fail("Expected IllegalArgumentException"); } catch (IllegalArgumentException e) { assertEquals("Max must be greater than min", e.getMessage()); } } @Test public void testGetRandomNumberMinAndMaxValid() { int min = 10; int max = 20; int randomNumber = getRandomNumber(min, max); assertTrue(randomNumber >= min && randomNumber <= max); } @Test public void testGetRandomNumberMultipleTimes() { Random random = new Random(); int min = 1; int max = 10; int numberOfTests = 10000; for (int i = 0; i < numberOfTests; i++) { int randomNumber1 = getRandomNumber(min, max); int randomNumber2 = getRandomNumber(min, max); assertTrue(randomNumber1 != randomNumber2); } } } Random input: Conclusion Integrating Ollama locally ensures data privacy and saves costs compared to closed-source commercial LLM models. Closed-source commercial LLM models are powerful, but this approach provides an alternative if open-source models can perform simple tasks. You can find the source code on GitHub.

By Amol Gote CORE

Introduction to Kata Containers Kata Containers is an open-source project designed to deliver a secure container runtime environment by utilizing the virtualization layer provided by the server instance. Unlike traditional containers, Kata containers run within lightweight virtual machines (VMs) created using virtualization capabilities. This approach ensures robust isolation between the host operating system (OS) and the containers, making them a powerful choice for scenarios demanding heightened security. The Core Difference: Kata vs. Conventional Containers Conventional containers are software packages that share the host OS kernel for isolated execution. They achieve isolation using Linux technologies such as network namespaces and c-groups. In contrast, Kata containers add an extra layer of isolation through virtualization. Each container runs within its own lightweight VM, complete with its own OS and kernel. This architecture significantly enhances security by creating a stronger separation between the container and the host system. FeatureTraditional ContainersKATA CONTAINERS Isolation Shares host kernel Dedicated VM and kernel Security Limited to namespaces and cgroups Strong isolation via VMs Performance High performance, low overhead Slightly higher overhead Attack Surface Broader due to shared kernel Narrower with kernel isolation Architectural Overview Kata containers leverage a unique architecture that combines the lightweight nature of containers with the strong isolation provided by virtual machines (VMs). Image courtesy of https://katacontainers.io/ Core Components Kata runtime. Acts as the interface between the container ecosystem (e.g., Docker, Kubernetes) and the underlying hypervisor. Implements the Open Container Initiative (OCI) runtime specification, ensuring compatibility with container orchestration tools. Responsible for launching a lightweight VM to encapsulate the container.Lightweight hypervisor. Provides the virtualization layer, ensuring strong isolation for the container by creating a dedicated VM. Optimized for speed and reduced overhead compared to traditional hypervisors.Guest kernel. Each container (or pod) runs inside its own VM with a dedicated kernel. Ensures complete isolation from the host and other containers, reducing the attack surface.Agent. A small process running inside the VM that manages the container lifecycle. Facilitates communication between the Kata Runtime (on the host) and the container workload inside the VM. Executes commands like starting, stopping, and monitoring the container. Interaction Workflow Request initiation. A container creation request (e.g., docker run or a Kubernetes pod) triggers the Kata Runtime.VM creation. The runtime communicates with the hypervisor to start a lightweight VM. A minimal guest kernel is loaded into the VM.Container setup. The runtime injects the container image into the VM and starts it using the agent.Lifecycle management. The agent within the VM handles container commands, such as starting processes, networking setup, and monitoring. Benefits of Kata Containers Below are some advantages of using Kata containers for secured containerized workflows. Strong isolation. Each container runs inside its own lightweight VM with a dedicated kernel. This helps prevent container escape vulnerabilities and provides multi-tenant workload isolation.Enhanced security. They reduce attack surfaces by isolating workloads at the kernel level. Compatibility with container ecosystems. They are fully compliant with OCI runtime standards.Lightweight virtualization. Each container uses lightweight hypervisors (e.g., QEMU, Firecracker) for minimal overhead.Multi-Tenant workload isolation. Kata containers prevent data leakage or interference between tenants.Improved compliance. They meet stringent compliance and regulatory requirements (e.g., GDPR, HIPAA).Open-source and community-driven. They are backed by an active open-source community under the Open Infrastructure Foundation (OpenStack). Why Kata Containers Are Not Always the Right Fit While they offer strong security and isolation, the unique architecture of Kata containers may not suit every workload or environment. Below are some examples. Applications requiring maximum performance. The VM-based architecture introduces additional overhead compared to traditional containers. Resource-constrained environments. Lightweight hypervisors consume more resources (e.g., memory and CPU) than traditional container runtimes.Lightweight, temporary workloads. For ephemeral or short-lived workloads, the overhead of starting a VM outweighs the security benefits.Cost-sensitive deployments. The increased resource consumption (e.g., memory and CPU) for each container results in higher infrastructure costs compared to traditional containers.Applications needing direct access to host hardware. The VM abstraction layer limits direct access to host devices like GPUs or specialized hardware.Environments with heavy networking requirements. Network performance can be slightly slower because of the added virtualization layer. Scenarios without compliance or security needs. If compliance and security isolation are not critical, traditional containers offer simpler and faster solutions. Conclusion Kata containers combine the speed and efficiency of Kubernetes pods with the strong security of virtual machines. They provide enhanced isolation, making them ideal for multi-tenant environments, sensitive applications, and compliance-driven workloads. They ultimately enable organizations to achieve operational efficiency and security by bridging the gap between containers and VMs. Stay tuned for Part 2, where we’ll provide a step-by-step guide to deploying and validating Kata containers in a practical cloud environment.

Salesforce’s Lightning Web Component (LWC) is a modern UI framework that developers use to create custom pages and functionalities on the Salesforce Lightning platform. While LWC allows developers to build powerful and interactive user interfaces, it often requires back-end integration to fetch or update data from Salesforce. This back-end support is provided by Apex, Salesforce's server-side programming language. In this article, we will explore three common methods for integrating LWC with Apex: Using the Wire Property with an Apex methodUsing the Wire Function with an Apex methodCalling an Apex method imperatively To illustrate these concepts, we’ll walk through a simple example: a Lead Search Page, where users input a phone number to search for leads in Salesforce. Wire Property With Apex Let us begin by creating an Apex method, as shown below. There are two key requirements for the Apex method to be wired to an LWC: The method should have an @AuraEnabled annotationThe method must be marked as cacheable = true The cacheable = true setting indicates that the method is only fetching data and not making any modifications. This improves performance by enabling the client side to use cached data, reducing unnecessary trips to the server trips when data is already available. Java public with sharing class LeadsController { @AuraEnabled (cacheable=true) public static list<Lead> getLeadsByPhone(String phone) { try { list<Lead> leads = [SELECT id, FirstName, LastName,Phone from Lead where phone =: phone]; return leads; } catch (Exception e) { throw new AuraHandledException(e.getMessage()); } } } The next step is to import the Apex method to LWC, as shown below. JavaScript import getLeadsByPhone from "@salesforce/apex/LeadsController.getLeadsByPhone"; Once the method is imported, wire that method to a property, as shown below. JavaScript import { LightningElement,wire,track } from 'lwc'; import getLeadsByPhone from "@salesforce/apex/LeadsController.getLeadsByPhone"; export default class Leads_search extends LightningElement { @track searchPhone; @wire(getLeadsByPhone,{phone:'$searchPhone'})leads; handleInputFieldChange(event){ this.searchPhone = event.target.value; } } In this example, the getLeadsByPhone method is wired to the leads property. The leads property will now have the following data structure: JavaScript { data: [/* array of leads */], error: {/* error information */} } leads.data can then be used in the HTML, as shown below. HTML <template> <lightning-card title="Leads Search"> <div class="slds-var-p-around_medium"> <lightning-input type="search" label="Phone Number" onchange={handleInputFieldChange} > </lightning-input> <template if:true={leads.data}> <lightning-datatable data={leads.data} key-field="Id" columns={columns} hide-checkbox-column="true" ></lightning-datatable> </template> </div> </lightning-card> </template> Wire Function With Apex Instead of wiring an Apex method to a property, you can wire it to a function to allow for further processing of the data. Let us take a look at how this wiring works. Going by the same example, as shown below, the getLeadsByPhone is now wired to a function called leadList where we calculate the size of the array returned by the Apex method along with assigning the array to a defined variable called leads. JavaScript import { LightningElement,wire,track } from 'lwc'; import getLeadsByPhone from "@salesforce/apex/LeadsController.getLeadsByPhone"; export default class Leads_search extends LightningElement { @track searchPhone; @track leads; @track screenMessage; @wire(getLeadsByPhone,{phone:'$searchPhone'}) leadList({data,error}){ if(data){ this.screenMessage = data.length; this.leads = data; }else if(error){ console.log(error); } } columns = [ { label: "FirstName", fieldName: "FirstName"}, { label: "LastName", fieldName: "LastName" }, { label: "Phone", fieldName: "Phone" } ]; handleInputFieldChange(event){ this.searchPhone = event.target.value; } } Now, we will retrieve the lead details from {leads} instead of {leads.data}. The calculated array size will be displayed on the screen. HTML <template> <lightning-card title="Leads Search"> <div class="slds-var-p-around_medium"> <lightning-input type="search" label="Phone Number" onchange={handleInputFieldChange} > </lightning-input> <template if:true={leads}> <div>{screenMessage} Leads</div> <lightning-datatable data={leads} key-field="Id" columns={columns} hide-checkbox-column="true" ></lightning-datatable> </template> </div> </lightning-card> </template> Here’s what the above component looks like in Salesforce. Imperative Method With Apex Apex methods can also be called directly from LWC, typically in response to a user action, such as a button click. This approach gives you greater control over when the method is invoked. The requirement for the Apex method remains the same. You still use the @AuraEnabled annotation. However, in this case, setting cacheable = true is optional. In our example, we will add a 'Search' button, which, when clicked, will display the lead list. HTML is updated as shown below. HTML <template> <lightning-card title="Leads Search"> <div class="slds-var-p-around_medium"> <lightning-input type="search" label="Phone Number" onchange={handleInputFieldChange} > </lightning-input> <lightning-button label="Search" onclick={handleSearch} variant="brand"></lightning-button> <template if:true={leads}> <div>{screenMessage} Leads</div> <lightning-datatable data={leads} key-field="Id" columns={columns} hide-checkbox-column="true" ></lightning-datatable> </template> </div> </lightning-card> </template> The search button is introduced, and when the button is clicked, the handleSearch method is called. Now, let us take a look at what an imperative call looks like in Javascript. The button method handleSearcch is introduced, which calls the Apex method and the returned result is handled as shown below. JavaScript import { LightningElement,wire,track } from 'lwc'; import getLeadsByPhone from "@salesforce/apex/LeadsController.getLeadsByPhone"; export default class Leads_search extends LightningElement { @track searchPhone; @track leads; @track screenMessage; columns = [ { label: "FirstName", fieldName: "FirstName"}, { label: "LastName", fieldName: "LastName" }, { label: "Phone", fieldName: "Phone" } ]; handleInputFieldChange(event){ this.searchPhone = event.target.value; } handleSearch(){ getLeadsByPhone({phone: this.searchPhone}) .then(result => { this.leads = result; this.screenMessage = result.length; }) .catch(error => { this.leads = undefined; this.screenMessage = "No Leads Found"; }); } } Here’s what the updated component looks like in Salesforce. Wire vs. Imperative Comparison Wire Calls Imperative Calls Reactivity Wire calls have an automatic connection with the server and are triggered automatically when variable changes are tracked. The Apex method is called explicitly in response to events such as a button click. Caching Data is stored in a reactive variable, which is automatically refreshed only when the tracked variable changes. They have no in-built caching. Direct Control There is no direct control over when the Apex method is invoked. There is full control of when and how the Apex method is invoked. Complexity The wired Apex method supports only simple queries. Imperative apex methods support complex queries, making them ideal for complex use cases. DML Operations They cannot handle DML operations like insert, update, or delete, as wire calls are ready-only. They are suitable for DML operations such as insert, update, and delete. Error Handling Automatic error handling and errors are returned through the wire result. Requires manual error handling and custom logic to handle exceptions and responses. Unsupported Objects They cannot be used with objects not supported by the User Interface API, such as Task and Event. They can handle objects not supported by the User Interface API, including Task and Event. Conclusion This article covers the three ways to integrate Apex with your Lightning components. While the @wire method provides reactivity and in-built caching, imperative calls give you more control over when and how Apex methods are invoked. The best approach depends on your specific use cases. However, if possible, it is recommended to use @wire as it allows you to take advantage of performance benefits like in-built caching and reactivity.

Software engineers must develop applications that are not only functional but also scalable, resilient, and globally distributed. This is where cloud computing plays a crucial role. Cloud platforms provide the foundation to build scalable microservices, ensuring high availability, efficient resource management, and seamless integration with modern technologies. The Importance of the Cloud in Your Software Engineering Career Cloud computing is no longer optional but a necessity for modern software engineers. Here’s why: Scalability. The cloud enables applications to handle growing user demands without requiring massive infrastructure investments.Resilience. Distributed systems in the cloud can tolerate failures, ensuring service continuity.Global reach. Cloud providers offer global data centers, reducing latency and improving user experience worldwide.Cost efficiency. Pay-as-you-go pricing models allow businesses to optimize costs using only the needed resources.Security and compliance. Cloud providers implement best practices, making it easier to comply with industry regulations. By mastering cloud technologies, software engineers open doors to career opportunities in DevOps, site reliability engineering, and cloud-native application development. Companies increasingly seek engineers who understand cloud architectures, containerization, and serverless computing. Exploring Jakarta NoSQL, Helidon, and Cloud-Based Microservices To develop scalable microservices, developers need robust frameworks and cloud-based infrastructure. Three key technologies play a crucial role in achieving this: Jakarta NoSQL With Jakarta Data Jakarta NoSQL is a specification that simplifies the integration of Java applications with NoSQL databases, allowing developers to work with repositories and object-mapping abstractions. Jakarta Data enhances this by standardizing data access, providing a more seamless experience when managing persistence operations. In Domain-Driven Design (DDD), repositories play a crucial role by providing an abstraction layer between the domain model and database interactions, ensuring the domain remains independent of infrastructure concerns. Helidon Helidon is a lightweight Java framework optimized for microservices. It offers two development models — Helidon SE, a reactive, functional approach, and Helidon MP, which follows Jakarta EE and MicroProfile standards. This flexibility makes it easier to build efficient and scalable cloud-native applications. In the DDD context, Helidon provides the tools to implement service layers, handling use cases while ensuring separation between the application layer and domain logic. Oracle Cloud Infrastructure (OCI) Oracle Cloud provides a resilient, high-performance environment for deploying microservices. With support for OCI Containers, API management, and global database solutions like Oracle NoSQL, developers can build and scale applications efficiently while ensuring low-latency access to data worldwide. Cloud infrastructure aligns with the strategic design principles in DDD by enabling bounded contexts to be deployed independently, fostering modular and scalable architectures. One key advantage of cloud-based microservices is integrating with globally distributed databases, ensuring high availability and scalability. This approach empowers developers to optimize data read and write performance while maintaining low-latency operations. Developers can explore practical applications that integrate these technologies in real-world scenarios for a more profound, hands-on experience. Understanding Domain-Driven Design Concepts in the Microservice This section introduces a Book Management Catalog Microservice, a sample project demonstrating how to implement microservices using Java, Helidon, Jakarta NoSQL, and Oracle Cloud. This microservice exemplifies how Domain-Driven Design (DDD) principles can be applied to real-world applications. Why Domain-Driven Design (DDD)? DDD is essential when building complex software systems, as it helps developers align their models with business requirements. DDD enhances maintainability, scalability, and adaptability by structuring software around the domain. In this microservice, DDD principles ensure clear separation between different architectural layers, making the application more modular and straightforward to evolve. Entity: Book In DDD, an Entity is an object defined by its identity rather than its attributes. The Book entity represents a distinct domain concept and is uniquely identified by its id. The Book entity persists in a NoSQL database, ensuring a scalable and flexible data structure. Java @Entity public class Book { @Id private String id = java.util.UUID.randomUUID().toString(); @Column private String title; @Column private BookGenre genre; @Column private int publicationYear; @Column private String author; @Column private List<String> tags; } Value Object: BookGenre A Value Object in DDD represents a concept in the domain that has no identity and is immutable. The BookGenre enum is a classic example of a value object, as it encapsulates book categories without requiring unique identification. Java public enum BookGenre { ACTION, COMEDY, DRAMA, HORROR, SCIENCE_FICTION, FANTASY, ROMANCE, THRILLER, FICTION, DYSTOPIAN } Repository: Managing Persistence The repository pattern in DDD abstracts persistence, providing a mechanism to retrieve domain objects while keeping the domain model free from infrastructure concerns. Jakarta Data simplifies the implementation of repositories, ensuring that data access is clean and structured. Java @Repository public interface BookRepository extends BasicRepository<Book, String> { } Service Layer: Application Logic The service layer orchestrates use cases and provides an interface for interactions with the domain model. This layer remains distinct from domain logic and is responsible for handling application-specific operations. Java @ApplicationScoped public class BookService { private final BookRepository bookRepository; private final BookMapper bookMapper; @Inject public BookService(@Database(DatabaseType.DOCUMENT) BookRepository bookRepository, BookMapper bookMapper) { this.bookRepository = bookRepository; this.bookMapper = bookMapper; } } Conclusion By leveraging Jakarta NoSQL, Helidon, and Oracle Cloud, developers can build resilient, scalable microservices that align with Domain-Driven Design principles. Entities, value objects, repositories, and service layers contribute to a well-structured, maintainable application. Cloud infrastructure further enhances these capabilities, allowing microservices to be deployed globally with high availability. To further explore these concepts in practice, consider this workshop on Oracle LiveLabs, where you can gain hands-on experience with Jakarta NoSQL, Helidon, and Oracle Cloud while building a real-world microservice.

By Otavio Santana CORE

First, we will see what Redis is and its usage, as well as why it is suitable for modern complex microservice applications. We will talk about how Redis supports storing multiple data formats for different purposes through its modules. Next, we will see how Redis, as an in-memory database, can persist data and recover from data loss. We’ll also talk about how Redis optimizes memory storage costs using Redis on Flash. Then, we will see very interesting use cases of scaling Redis and replicating it across multiple geographic regions. Finally, since one of the most popular platforms for running micro-services is Kubernetes, and since running stateful applications in Kubernetes is a bit challenging, we will see how you can easily run Redis on Kubernetes. What Is Redis? Redis, which actually stands for Remote Dictionary Server, is an in-memory database. So many people have used it as a cache on top of other databases to improve the application performance. However, what many people don’t know is that Redis is a fully fledged primary database that can be used to store and persist multiple data formats for complex applications. Complex Social Media Application Example Let’s look at a common setup for a microservices application. Let’s say we have a complex social media application with millions of users. And let’s say our microservices application uses a relational database like MySQL to store the data. In addition, because we are collecting tons of data daily, we have an Elasticsearch database for fast filtering and searching the data. Now, the users are all connected to each other, so we need a graph database to represent these connections. Plus, our application has a lot of media content that users share with each other daily, and for that, we have a document database. Finally, for better application performance, we have a cache service that caches data from other databases and makes it accessible faster. Now, it’s obvious that this is a pretty complex setup. Let’s see what the challenges of this setup are: 1. Deployment and Maintenance All these data services need to be deployed, run, and maintained. This means your team needs to have some kind of knowledge of how to operate all these data services. 2. Scaling and Infrastructure Requirements For high availability and better performance, you would want to scale your services. Each of these data services scales differently and has different infrastructure requirements, and that could be an additional challenge. So overall, using multiple data services for your application increases the effort of maintaining your whole application setup. 3. Cloud Costs Of course, as an easier alternative to running and managing the services yourself, you can use the managed data services from cloud providers. But this could be very expensive because, on cloud platforms, you pay for each managed data service separately. 4. Development Complexity On the development side, your application code also gets pretty complex because you need to talk to multiple data services. For each service, you would need a separate connector and logic. This makes testing your applications also quite challenging. 5. Higher Latency The more number of services that talk to each other, the higher the latency. Even though each service may be fast on its own, each connection step between the services or each network hop will add some latency to your application. Why Redis Simplifies This Complexity In comparison with a multi-modal database like Redis, you resolve most of these challenges: Single data service. You run and maintain just one data service. So your application also needs to talk to a single data store, which means only one programmatic interface for that data service.Reduced latency. Latency will be reduced by going to a single data endpoint and eliminating several internal network hops.Multiple data types in one. Having one database like Redis that allows you to store different types of data (i.e., multiple types of databases in one) as well as act as a cache solves such challenges. How Redis Supports Multiple Data Formats So, let’s see how Redis actually works. First of all, how does Redis support multiple data formats in a single database? Redis Core and Modules The way it works is that you have Redis core, which is a key-value store that already supports storing multiple types of data. Then, you can extend that core with what’s called modules for different data types, which your application needs for different purposes. For example: RedisSearch for search functionality (like Elasticsearch)RedisGraph for graph data storage A great thing about this is that it’s modular. These different types of database functionalities are not tightly integrated into one database as in many other multi-modal databases, but rather, you can pick and choose exactly which data service functionality you need for your application and then basically add that module. Built-In Caching And, of course, when using Redis as a primary database, you don’t need an additional cache because you have that automatically out of the box with Redis. That means, again, less complexity in your application because you don’t need to implement the logic for managing, populating, and invalidating the cache. High Performance and Faster Testing Finally, as an in-memory database, Redis is super fast and performant, which, of course, makes the application itself faster. In addition, it also makes running the application tests way faster, as well, because Redis doesn’t need a schema like other databases. So it doesn’t need time to initialize the database, build the schema, and so on before running the tests. You can start with an empty Redis database every time and generate data for tests as you need. Fast tests can really increase your development productivity. Data Persistence in Redis We understood how Redis works and all its benefits. But at this point, you may be wondering: How can an in-memory database persist data? Because if the Redis process or the server on which Redis is running fails, all the data in memory is gone, right? And if I lose the data, how can I recover it? So basically, how can I be confident that my data is safe? The simplest way to have data backups is by replicating Redis. So, if the Redis master instance goes down, the replicas will still be running and have all the data. If you have a replicated Redis, the replicas will have the data. But of course, if all the Redis instances go down, you will lose the data because there will be no replica remaining. We need real persistence. Snapshot (RDB) Redis has multiple mechanisms for persisting the data and keeping the data safe. The first one is snapshots, which you can configure based on time, number of requests, etc. Snapshots of your data will be stored on a disk, which you can use to recover your data if the whole Redis database is gone. But note that you will lose the last minutes of data, because you usually do snapshotting every five minutes or an hour, depending on your needs. AOF (Append Only File) As an alternative, Redis uses something called AOF, which stands for Append Only File. In this case, every change is saved to the disk for persistence continuously. When restarting Redis or after an outage, redis will replay the Append Only File logs to rebuild the state. So, AOF is more durable but can be slower than snapshotting. Combination of Snapshots and AOF And, of course, you can also use a combination of both AOF and snapshots, where the append-only file is persisting data from memory to disk continuously, plus you have regular snapshots in between to save the data state in case you need to recover it. This means that even if the Redis database itself or the servers, the underlying infrastructure where Redis is running, all fail, you still have all your data safe and you can easily recreate and restart a new Redis database with all the data. Where Is This Persistent Storage? A very interesting question is, where is that persistent storage? So where is that disk that holds your snapshots and the append-only file logs located? Are they on the same servers where Redis is running? This question actually leads us to the trend or best practice of data persistence in cloud environments, which is that it’s always better to separate the servers that run your application and data services from the persistent storage that stores your data. With a specific example: If your applications and services run in the cloud on, let’s say, an AWS EC2 instance, you should use EBS or Elastic Block Storage to persist your data instead of storing them on the EC2 instance’s hard drive. Because if that EC2 instance dies, you won’t have access to any of its storage, whether it’s RAM or disk storage or whatever. So, if you want persistence and durability for your data, you must put your data outside the instances on an external network storage. As a result, by separating these two, if the server instance fails or if all the instances fail, you still have the disk and all the data on it unaffected. You just spin up other instances and take the data from the EBS, and that’s it. This makes your infrastructure way easier to manage because each server is equal; you don’t have any special servers with any special data or files on it. So you don’t care if you lose your whole infrastructure because you can just recreate a new one and pull the data from a separate storage, and you’re good to go again. Going back to the Redis example, the Redis service will be running on the servers and using server RAM to store the data, while the append-only file logs and snapshots will be persisted on a disk outside those servers, making your data more durable. Cost Optimization With Redis on Flash Now we know you can persist data with Redis for durability and recovery while using RAM or memory storage for great performance and speed. So the question you may have here is: Isn’t storing data in memory expensive? Because you would need more servers compared to a database that stores data on disk simply because memory is limited in size. There’s a trade-off between the cost and performance. Well, Redis actually has a way to optimize this using a service called Redis on Flash, which is part of Redis Enterprise. How Redis on Flash Works It’s a pretty simple concept, actually: Redis on Flash extends the RAM to the flash drive or SSD, where frequently used values are stored in RAM and the infrequently used ones are stored on SSD. So, for Redis, it’s just more RAM on the server. This means that Redis can use more of the underlying infrastructure or the underlying server resources by using both RAM and SSD drive to store the data, increasing the storage capacity on each server, and this way saving you infrastructure costs. Scaling Redis: Replication and Sharding We’ve talked about data storage for the Redis database and how it all works, including the best practices. Now another very interesting topic is how do we scale a Redis database? Replication and High Availability Let’s say my one Redis instance runs out of memory, so data becomes too large to hold in memory, or Redis becomes a bottleneck and can’t handle any more requests. In such a case, how do I increase the capacity and memory size of my Redis database? We have several options for that. First of all, Redis supports clustering, which means you can have a primary or master Redis instance that can be used to read and write data, and you can have multiple replicas of that primary instance for reading the data. This way, you can scale Redis to handle more requests and, in addition, increase the high availability of your database. If the master fails, one of the replicas can take over, and your Redis database can basically continue functioning without any issues. These replicas will all hold copies of the data of the primary instance. So, the more replicas you have, the more memory space you need. And one server may not have sufficient memory for all your replicas. Plus, if you have all the replicas on one server and that server fails, your whole Redis database is gone, and you will have downtime. Instead, you want to distribute these replicas among multiple nodes or servers. For example, your master instance will be on one node and two replicas on the other two nodes. Sharding for Larger Datasets Well, that seems good enough, but what if your data set grows too large to fit in memory on a single server? Plus, we have scaled the reads in the database, so all the requests basically just query the data, but our master instance is still alone and still has to handle all the writes. So, what is the solution here? For that, we use the concept of sharding, which is a general concept in databases and which Redis also supports. Sharding basically means that you take your complete data set and divide it into smaller chunks or subsets of data, where each shard is responsible for its own subset of data. That means instead of having one master instance that handles all the writes to the complete data set; you can split it into, let’s say, four shards, each of them responsible for reads and writes to a subset of the data. Each shard also needs less memory capacity because it just has a fourth of the data. This means you can distribute and run shards on smaller nodes and basically scale your cluster horizontally. And, of course, as your data set grows and as you need even more resources, you can re-shard your Redis database, which basically means you just split your data into even smaller chunks and create more shards. So having multiple nodes that run multiple replicas of Redis, which are all sharded, gives you a very performant, highly available Redis database that can handle many more requests without creating any bottlenecks. Now, I have to note here that this setup is great, but you would need to manage it yourself, do the scaling, add nodes, do the sharding, and then resharding, etc. For some teams that are more focused on application development and more business logic rather than running and maintaining data services, this could be a lot of unwanted effort. So, as an easier alternative, in Redis Enterprise, you get this kind of setup automatically because the scaling, sharding, and so on are all managed for you. Global Replication With Redis: Active-Active Deployment Let’s consider another interesting scenario for applications that need even higher availability and performance across multiple geographic locations. So let’s say we have this replicated, sharded Redis database cluster in one region, in the data center of London, Europe. But we have the two following use cases: Our users are geographically distributed, so they are accessing the application from all over the world. We want to distribute our application and data services globally, close to the users, to give our users better performance.If the complete data center in London, Europe, for example, goes down, we want an immediate switch-over to another data center so that the Redis service stays available. In other words, we want replicas of the whole Redis cluster in data centers in multiple geographic locations or regions. Multiple Redis Clusters Across Regions This means a single data should be replicated to many clusters spread across multiple regions, with each cluster being fully able to accept reads and writes. In that case, you would have multiple Redis clusters that will act as local Redis instances in each region, and the data will be synced across these geographically distributed clusters. This is a feature available in Redis Enterprise and is called active-active deployment because you have multiple active databases in different locations. With this setup, we’ll have lower latency for the users. And even if the Redis database in one region completely goes down, the other regions will be unaffected. If the connection or syncing between the regions breaks for a short time because of some network problem, for example, the Redis clusters in these regions can update the data independently, and once the connection is re-established, they can sync up those changes again. Conflict Resolution With CRDTs Now, of course, when you hear that, the first question that may pop up in your mind is: How does Redis resolve the changes in multiple regions to the same data set? So, if the same data changes in multiple regions, how does Redis make sure that data changes of any region aren’t lost and data is correctly synced, and how does it ensure data consistency? Specifically, Redis Enterprise uses a concept called CRDTs, which stands for conflict-free replicated Data types, and this concept is used to resolve any conflicts automatically at the database level and without any data loss. So basically, Redis itself has a mechanism for merging the changes that were made to the same data set from multiple sources in a way that none of the data changes are lost and any conflicts are properly resolved. And since, as you learned, Redis supports multiple data types, each data type uses its own data conflict resolution rules, which are the most optimal for that specific data type. Simply put, instead of just overriding the changes of one source and discarding all the others, all the parallel changes are kept and intelligently resolved. Again, this is automatically done for you with this active-active geo-replication feature, so you don’t need to worry about that. Running Redis in Kubernetes And the last topic I want to address with Redis is running Redis in Kubernetes. As I said, Redis is a great fit for complex micro-services that need to support multiple data types and that need an easy scaling of a database without worrying about data consistency. And we also know that the new standard for running microservices is the Kubernetes platform. So, running Redis in Kubernetes is a very interesting and common use case. So how does that work? Open Source Redis on Kubernetes With open-source Redis, you can deploy replicated Redis as a Helm chart or Kubernetes manifest files and, basically, using the replication and scaling rules that we already talked about, set up and run a highly available Redis database. The only difference would be that the hosts where Redis will run will be Kubernetes pods instead of, for example, EC2 instances or any other physical or virtual servers. But the same sharding, replicating, and scaling concepts apply here as well when you want to run a Redis cluster in Kubernetes, and you would basically have to manage that setup yourself. Redis Enterprise Operator However, as I mentioned, many teams don’t want to make the effort to maintain these third-party services because they would rather invest their time and resources in application development or other tasks. So, having an easier alternative is important here as well. Redis Enterprise has a managed Redis cluster, which you can deploy as a Kubernetes operator. If you don’t know operators, an operator in Kubernetes is basically a concept where you can bundle all the resources needed to operate a certain application or service so that you don’t have to manage it yourself or you don’t have to operate it yourself. Instead of a human operating a database, you basically have all this logic in an automated form to operate a database for you. Many databases have operators for Kubernetes, and each such operator has, of course, its own logic based on who wrote them and how they wrote them. The Redis Enterprise on Kubernetes operator specifically automates the deployment and configuration of the whole Redis database in your Kubernetes cluster. It also takes care of scaling, doing the backups, and recovering the Redis cluster if needed, etc. So, it takes over the complete operation of the Redis cluster inside the Kubernetes cluster. Conclusion I hope you learned a lot in this blog and that I was able to answer many of your questions. If you want to learn more similar technologies and concepts, then make sure to follow me because I write blogs regularly about AI, DevOps, and cloud technologies. Also, comment below if you have any questions regarding Redis or any new topic suggestions. And with that, thank you for reading, and see you in the next blog. Let’s connect on LinkedIn!

Generating ETL data pipelines using generative AI (GenAI) involves leveraging the capabilities of large language models to automatically create the code and logic for extracting, transforming, and loading data from various sources, significantly reducing manual coding efforts and accelerating pipeline development by allowing users to describe their desired data transformations in natural language prompts, which the AI then translates into executable code. What Is ETL Pipeline? Data pipelines are the hidden engines that keep modern businesses running smoothly. They quietly transport data from various sources to warehouses and lakes, where it can be stored and used for decision-making. These pipelines perform the essential task of moving and organizing data behind the scenes, rarely noticed — until something breaks down. The ETL (Extract, Transform, Load) process is central to these data pipelines, ensuring data is properly formatted, transformed, and loaded for use. However, ETL processes can face disruptions due to schema changes, data errors, or system limitations. This is where generative AI (GenAI) comes into play, adding intelligence and flexibility to the ETL process. By combining traditional data pipelines with the capabilities of AI, organizations can unlock new ways of automating and optimizing how data flows. In this article, we'll explore how GenAI is making ETL and data pipelines smarter, more efficient, and capable of adapting to ever-changing data requirements. ETL Data Pipelines Generation Issues The ETL process extracts data from various sources, transforms it into the correct format, and loads it into a database or data warehouse. This process allows businesses to organize their data so it's ready for analysis, reporting, and decision-making. However, despite its critical role, traditional ETL faces several challenges: Schema changes. When data structures or formats change (e.g., a new field is added or renamed), the ETL process can fail, often requiring manual intervention to fix.Data quality issues. Incorrect or missing data can cause processing errors, leading to data inconsistency or incomplete analysis.Scalability concerns. As data volumes grow, existing ETL systems can struggle to handle the load, causing delays or failures in the data pipeline.Error handling. If there is a hardware failure or a process error, data pipelines can break, often requiring time-consuming troubleshooting and resolution. With the growing complexity and volume of data, businesses need more advanced and resilient systems. That's where GenAI comes in, offering solutions that go beyond traditional ETL approaches. Key Aspects of Using GenAI for ETL Pipelines Code Generation GenAI can generate code snippets or complete ETL scripts based on user-defined data sources, desired transformations, and target destinations, including functions for data cleaning, filtering, aggregation, and more. Data Schema Understanding GenAI can automatically identify data structures and relationships by analyzing data samples and suggesting optimal data models and schema designs for the target database. Self-Updating Pipelines One of the most powerful features is the ability to automatically adapt pipelines to changes in source data schema by detecting new fields or modifications and updating the extraction and transformation logic accordingly. Data Quality Validation GenAI can generate data quality checks and validation rules based on historical data patterns and business requirements to ensure data integrity throughout the pipeline. How to Implement GenAI for ETL 1. Describe the Pipeline Clearly define the data sources, desired transformations, and target destinations using natural language prompts, providing details like specific columns, calculations, and data types. 2. Choose a GenAI Tool Select a suitable GenAI platform or tool that integrates with your preferred data engineering environment, considering factors like model capabilities, supported languages, and data privacy considerations. 3. Provide Data Samples If necessary, provide representative data samples to the AI model to enable a better understanding of data characteristics and potential transformations. 4. Generate Code Based on the prompts and data samples, the GenAI will generate the ETL code, including extraction queries, transformation logic, and loading statements. 5. Review and Refine While the generated code can be largely functional, manual review and fine-tuning may be needed to address specific edge cases or complex transformations. Benefits of Using GenAI for ETL One of the most exciting possibilities with GenAI is the ability to create self-updating ETL pipelines. These AI-powered systems can detect changes in data structures or schemas and automatically adjust the pipeline code to accommodate them. Increased Efficiency Significantly reduces development time by automating code generation for common ETL tasks. Improved Agility Enables faster adaptation to changing data sources and requirements by facilitating self-updating pipelines. Reduced Manual Effort Lessens the need for extensive manual coding and debugging, allowing data engineers to focus on more strategic tasks. Important Considerations Data Privacy Ensure that sensitive data is appropriately protected when using GenAI models, especially when working with large datasets. Model Accuracy Validate the generated code thoroughly and monitor performance to identify potential issues and refine the AI model as needed. Domain Expertise While GenAI can automate significant parts of ETL development, domain knowledge is still crucial to design effective pipelines and handle complex data transformations. Conclusion AI is introducing a new era of efficiency and adaptability in data management by making data pipelines self-updating, self-healing, and capable of advanced data aggregation and matching. From automating data cleanup to performing analyst-level tasks, generative AI promises to make data processes faster, more reliable, and easier to manage. While there are still challenges around privacy and security, advancements in secure AI deployment are making it possible to harness AI without sacrificing data integrity.

The cloud has proven to be a main enabler for large AI deployments because it provides AI-native APIs for rapid prototyping, elastic computing, and storage to address scaling problems. This article covers how to build and scale GenAI applications in the cloud. The Importance of the Cloud in GenAI The cloud is critical for contemporary GenAI applications because it can accommodate vast processing power, data storage, and distributed processes necessary for AI models. Traditional deployments often need more flexibility and performance to adapt to changing business requirements. Microsoft Azure, AWS, and Google Cloud are examples of cloud AI service providers. For example, Azure AI provides ready-to-utilize algorithms and models and the necessary infrastructural tools for building and expanding AI applications. In addition, GenAI projects that are cloud-based also benefit from the following advantages: Elastic provisioning: Resources are provisioned automatically or manually depending on business needs.Cost optimization: AI tools and AI-enabled tool configurations, plus automatic on-the-fly scaling can optimize operational costs. Not to mention the pay-as-you-go pricing model and hybrid cloud supported by large cloud providers. All of these improvements facilitate more focus on model development instead of hardware and infrastructural backing management.Integrated AI Services: Integration makes it possible to market faster by using pre-trained models and APIs or OpenAI and all advanced toolkits. Due to these advantages, the cloud is the core of the development of current generative AI, starting from the large language models (LLMs) to the multimodal AI systems. Data Preparation Any effective GenAI application relies on high-quality data. Training models on different, well-prepared datasets gives greater generalizability and resilience. Steps to Prepare Data Data collection and ingestion: This feature allows cataloging datasets in the data storage tool of your choice. It also allows automatic data flow from many sources with the help of automated ingestion pipelines.Data cleaning and transformation: Certain data applications assist in cleansing and shaping unprocessed data into meaningful, useful forms.Data annotation and governance: Annotating specific datasets necessary for certain GenAI models can be done using annotation tools or cloud services. The more ample and well-structured the training sets are will help widen the ‘temporal cycles’ that can fit the models. Best Practices for GenAI Data Preparation Data governance: Ensure security through strict data protection, access, and legislative compliance regulations.Cloud-native compliance: Apply policies with the technology provider of your choice for user compliance verification.Data protection: Protect data access and ensure compliance with applicable legislation through regulatory data protection measures. Ensure you have a wide range of compliance certifications, including but not limited to SOC, GDPR, and HIPAA, which promise improved management of sensitive data.Cloud-native security: Take advantage of the tool provider of your choice's pre-existing security aspects, if available, which assist in advanced threat prevention with its ongoing surveillance and assurance of meeting set standards. Fine-Tuning Models Major cloud services would provide all the necessary resources to train and fine-tune GenAI models, including resources that can easily be reconfigured. Pre-trained model : Time and cost are greatly spared when employing already trained models, such as OpenAI's GPT-4 or DALL-E. Cloud GPUs or TPUs and frameworks such as Hugging Face, all of which allow for the adaptation of these models.Distributed training: Certain machine learning tools come with distributed training capabilities that enable good scaling across multiple nodes on the cloud. Moreover, it might be important for all programs to seek solutions for the development and resolution of problems of ethical artificial intelligence. Legitimate concerns regarding bias and fairness in AI can be effectively addressed with these tools, which often provide insights into model behavior and the detection and mitigation of biases. GenAI Modeling Factors for Deployment at Scale The evaluation of GenAI models in the revolutionary setting is always preceded by analyzing the cost of scalability, latency, and maintenance of the systems. Hosting models: Some OpenAI model deployments are achieved through scalable endpoints meant for ultra-low latency high-volume inferencing. Their sophisticated load balancer and elastically scaling resources buffer ensure that service delivery is superb regardless of the dynamic load. Serverless architectures: Serverless computing can automatically create the appropriate scale without the need for operable cost, although no per-infrastructural management is required. CI/CD integrates well with machine learning models, allowing model re-training and testing deployment to pipelines to be automated. The built-in monitoring and rollback feature guarantees rapid updates without excessive risk, making it perfect for managing highly available and reliable AI systems. Inference and Real-World Applications Inference, or the outputs produced from trained models, must be made while considering the aspects of latency, throughput, and cost. Considerations for Real-Time Inference Try using quantization or model pruning optimization techniques wherever possible to reduce the inference time. Be sure to employ managed inference services. Real-World Use Cases Predictive analytics: Knowing different patterns and facts using analytical methods drastically improves finance, health care, and logistics decisions.Automated content creation: Content generation employs AI to generate written content for various purposes, including creative writing, marketing, or product details. Challenges of Using GenAI Though GenAI offers promise, efforts at scaling its applications in the cloud have difficulties, including: Cost of infrastructure: Failure to properly understand the infrastructure requirements can lead to over-provisioning of resources or waste of vital infrastructure. Load testing and careful estimating of future demand are essential.Interdisciplinary collaboration: Even a functioning prototype often requires constructing and integrating cross-functional teams with technical and domain knowledge.Business alignment: Each model must be designed to solve so that value can be derived for each business. Modeling boosts development when data scientists, product management, and other stakeholders begin working together. Conclusion GenAI, when paired with cloud technology, provides an unparalleled possibility for innovation and scale. Organizations may overcome scaling problems by embracing the cloud's flexibility, enhanced capabilities, and cost-effectiveness, allowing GenAI to reach its disruptive promise.

By Aravind Nuthalapati CORE

Microservices and containers are revolutionizing how modern applications are built, deployed, and managed in the cloud. However, developing and operating microservices can introduce significant complexity, often requiring developers to spend valuable time on cross-cutting concerns like service discovery, state management, and observability. Dapr, or Distributed Application Runtime, is an open-source runtime for building microservices on cloud and edge environments. It provides platform-agnostic building blocks like service discovery, state management, pub/sub messaging, and observability out of the box. Dapr moved to the graduated maturity level of CNCF (Cloud Native Computing Foundation) and is currently used by many enterprises. When combined with Amazon Elastic Kubernetes Service (Amazon EKS), a managed Kubernetes service from AWS, Dapr can accelerate the adoption of microservices and containers, enabling developers to focus on writing business logic without worrying about infrastructure plumbing. Amazon EKS makes managing Kubernetes clusters easy, enabling effortless scaling as workloads change. In this blog post, we'll explore how Dapr simplifies microservices development on Amazon EKS. We'll start by diving into two essential building blocks: service invocation and state management. Service Invocation Seamless and reliable communication between microservices is crucial. However, developers often struggle with complex tasks like service discovery, standardizing APIs, securing communication channels, handling failures gracefully, and implementing observability. With Dapr's service invocation, these problems become a thing of the past. Your services can effortlessly communicate with each other using industry-standard protocols like gRPC and HTTP/HTTPS. Service invocation handles all the heavy lifting, from service registration and discovery to request retries, encryption, access control, and distributed tracing. State Management Dapr's state management building block simplifies the way developers work with the state in their applications. It provides a consistent API for storing and retrieving state data, regardless of the underlying state store (e.g., Redis, AWS DynamoDB, Azure Cosmos DB). This abstraction enables developers to build stateful applications without worrying about the complexities of managing and scaling state stores. Prerequisites In order to follow along this post, you should have the following: An AWS account. If you don’t have one, you can sign up for one.An IAM user with proper permissions. The IAM security principal that you're using must have permission to work with Amazon EKS IAM roles, service-linked roles, AWS CloudFormation, a VPC, and related resources. For more information, see Actions, resources, and condition keys for Amazon Elastic Container Service for Kubernetes and Using service-linked roles in the AWS Identity and Access Management User Guide. Application Architecture In the diagram below, we have two microservices: a Python app and a Node.js app. The Python app generates order data and invokes the /neworder endpoint exposed by the Node.js app. The Node.js app writes the incoming order data to a state store (in this case, Amazon ElastiCache) and returns an order ID to the Python app as a response. By leveraging Dapr's service invocation building block, the Python app can seamlessly communicate with the Node.js app without worrying about service discovery, API standardization, communication channel security, failure handling, or observability. It implements mTLS to provide secure service-to-service communication. Dapr handles these cross-cutting concerns, allowing developers to focus on writing the core business logic. Additionally, Dapr's state management building block simplifies how the Node.js app interacts with the state store (Amazon ElastiCache). Dapr provides a consistent API for storing and retrieving state data, abstracting away the complexities of managing and scaling the underlying state store. This abstraction enables developers to build stateful applications without worrying about the intricacies of state store management. The Amazon EKS cluster hosts a namespace called dapr-system, which contains the Dapr control plane components. The dapr-sidecar-injector automatically injects a Dapr runtime into the pods of Dapr-enabled microservices. Service Invocation Steps The order generator service (Python app) invokes the Node app’s method, /neworder. This request is sent to the local Dapr sidecar, which is running in the same pod as the Python app. Dapr resolves the target app using the Amazon EKS cluster’s DNS provider and sends the request to the Node app’s sidecar.The Node app’s sidecar then sends the request to the Node app microservice.Node app then writes the order ID received from the Python app to Amazon ElasticCache.The node app sends the response to its local Dapr sidecar.Node app’s sidecar forwards the response to the Python app’s Dapr sidecar. Python app side car returns the response to the Python app, which had initiated the request to the Node app's method /neworder. Deployment Steps Create and Confirm an EKS Cluster To set up an Amazon EKS (Elastic Kubernetes Service) cluster, you'll need to follow several steps. Here's a high-level overview of the process: Prerequisites Install and configure the AWS CLIInstall eksctl, kubectl, and AWS IAM Authenticator 1. Create an EKS cluster. Use eksctl to create a basic cluster with a command like: Shell eksctl create cluster --name my-cluster --region us-west-2 --node-type t3.medium --nodes 3 2. Configure kubectl. Update your kubeconfig to connect to the new cluster: Shell aws eks update-kubeconfig --name my-cluster --region us-west-2 3. Verify the cluster. Check if your nodes are ready: Shell kubectl get nodes Install DAPR on Your EKS cluster 1. Install DAPR CLI: Shell wget -q https://raw.githubusercontent.com/dapr/cli/master/install/install.sh -O - | /bin/bash 2. Verify installation: Shell dapr -h 3. Install DAPR and validate: Shell dapr init -k --dev dapr status -k The Dapr components statestore and pubsub are created in the default namespace. You can check it by using the command below: Shell dapr components -k Configure Amazon ElastiCache as Your Dapr StateStore Create Amazon ElastiCache to store the state for the microservice. In this example, we are using ElastiCache serverless, which quickly creates a cache that automatically scales to meet application traffic demands with no servers to manage. Configure the security group of the ElastiCache to allow connections from your EKS cluster. For the sake of simplicity, keep it in the same VPC as your EKS cluster. Take note of the cache endpoint, which we will need for the subsequent steps. Running a Sample Application 1. Clone the Git repo of the sample application: Shell git clone https://github.com/dapr/quickstarts.git 2. Create redis-state.yaml and provide an Amazon ElasticCache endpoint for redisHost: YAML apiVersion: dapr.io/v1alpha1 kind: Component metadata: name: statestore namespace: default spec: type: state.redis version: v1 metadata: - name: redisHost value: redisdaprd-7rr0vd.serverless.use1.cache.amazonaws.com:6379 - name: enableTLS value: true Apply yaml configuration for state store component using kubectl. Shell kubectl apply -f redis-state.yaml 3. Deploy microservices with the sidecar. For the microservice node app, navigate to the /quickstarts/tutorials/hello-kubernetes/deploy/node.yaml file and you will notice the below annotations. It tells the Dapr control plane to inject a sidecar and also assigns a name to the Dapr application. YAML annotations: dapr.io/enabled: "true" dapr.io/app-id: "nodeapp" dapr.io/app-port: "3000" Add an annotation service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" in node.yaml to create AWS ELB. YAML kind: Service apiVersion: v1 metadata: name: nodeapp annotations: service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" labels: app: node spec: selector: app: node ports: - protocol: TCP port: 80 targetPort: 3000 type: LoadBalancer Deploy the node app using kubectl. Navigate to the directory /quickstarts/tutorials/hello-kubernetes/deploy and execute the below command. Shell kubectl apply -f node.yaml Obtain the AWS NLB, which appears under External IP, on the output of the below command. Shell kubectl get svc nodeapp http://k8s-default-nodeapp-3a173e0d55-f7b14bedf0c4dd8.elb.us-east-1.amazonaws.com Navigate to the /quickstarts/tutorials/hello-kubernetes directory, which has sample.json file to execute the below step. Shell curl --request POST --data "@sample.json" --header Content-Type:application/json http://k8s-default-nodeapp-3a173e0d55-f14bedff0c4dd8.elb.us-east-1.amazonaws.com/neworder You can verify the output by accessing /order endpoint using the load balancer in a browser. Plain Text http://k8s-default-nodeapp-3a173e0d55-f7b14bedff0c4dd8.elb.us-east-1.amazonaws.com/order You will see the output as {“OrderId”:“42”} Next, deploy the second microservice Python app, which has a business logic to generate a new order ID every second and invoke the Node app’s method /neworder. Navigate to the directory /quickstarts/tutorials/hello-kubernetes/deploy and execute the below command. Shell kubectl apply -f python.yaml 4. Validating and testing your application deployment. Now that we have both the microservices deployed. The Python app is generating orders and invoking /neworder as evident from the logs below. Shell kubectl logs --selector=app=python -c daprd --tail=-1 SystemVerilog time="2024-03-07T12:43:11.556356346Z" level=info msg="HTTP API Called" app_id=pythonapp instance=pythonapp-974db9877-dljtw method="POST /neworder" scope=dapr.runtime.http-info type=log useragent=python-requests/2.31.0 ver=1.12.5 time="2024-03-07T12:43:12.563193147Z" level=info msg="HTTP API Called" app_id=pythonapp instance=pythonapp-974db9877-dljtw method="POST /neworder" scope=dapr.runtime.http-info type=log useragent=python-requests/2.31.0 ver=1.12.5 We can see that the Node app is receiving the requests and writing to the state store Amazon ElasticCache in our example. Shell kubectl logs —selector=app=node -c node —tail=-1 SystemVerilog Got a new order! Order ID: 367 Successfully persisted state for Order ID: 367 Got a new order! Order ID: 368 Successfully persisted state for Order ID: 368 Got a new order! Order ID: 369 Successfully persisted state for Order ID: 369 In order to confirm whether the data is persisted in Amazon ElasticCache we access the endpoint /order below. It returns the latest order ID, which was generated by the Python app. Plain Text http://k8s-default-nodeapp-3a173e0d55-f7b14beff0c4dd8.elb.us-east-1.amazonaws.com/order You will see an output with the most recent order as {“OrderId”:“370”}. Clean up Run the below command to delete the deployments Node app and Python app along with the state store component. Navigate to the /quickstarts/tutorials/hello-kubernetes/deploy directory to execute the below command. YAML kubectl delete -f node.yaml kubectl delete -f python.yaml You can tear down your EKS cluster using the eksctl command and delete Amazon ElastiCache. Navigate to the directory that has the cluster.yaml file used to create the cluster in the first step. Shell eksctl delete cluster -f cluster.yaml Conclusion Dapr and Amazon EKS form a powerful alliance for microservices development. Dapr simplifies cross-cutting concerns, while EKS manages Kubernetes infrastructure, allowing developers to focus on core business logic and boost productivity. This combination accelerates the creation of scalable, resilient, and observable applications, significantly reducing operational overhead. It's an ideal foundation for your microservices journey. Watch for upcoming posts exploring Dapr and EKS's capabilities in distributed tracing and observability, offering deeper insights and best practices.