DZone Spotlight

In part one of this two-part series, we looked at how walletless dApps smooth the web3 user experience by abstracting away the complexities of blockchains and wallets. Thanks to account abstraction from Flow and the Flow Wallet API, we can easily build walletless dApps that enable users to sign up with credentials that they're accustomed to using (such as social logins or email accounts). We began our walkthrough by building the backend of our walletless dApp. Here in part two, we'll wrap up our walkthrough by building the front end. Here we go! Create a New Next.js Application Let's use the Next.js framework so we have the frontend and backend in one application. On our local machine, we will use create-next-app to bootstrap our application. This will create a new folder for our Next.js application. We run the following command: Shell $ npx create-next-app flow_walletless_app Some options will appear; you can mark them as follows (or as you prefer!). Make sure to choose No for using Tailwind CSS and the App Router. This way, your folder structure and style references will match what I demo in the rest of this tutorial. Shell ✔ Would you like to use TypeScript with this project? ... Yes ✔ Would you like to use ESLint with this project? ... No ✔ Would you like to use Tailwind CSS with this project? ... No <-- IMPORTANT ✔ Would you like to use `src/` directory with this project? ... No ✔ Use App Router (recommended)? ... No <-- IMPORTANT ✔ Would you like to customize the default import alias? ... No Start the development server. Shell $ npm run dev The application will run on port 3001 because the default port (3000) is occupied by our wallet API running through Docker. Set Up Prisma for Backend User Management We will use the Prisma library as an ORM to manage our database. When a user logs in, we store their information in a database at a user entity. This contains the user's email, token, Flow address, and other information. The first step is to install the Prisma dependencies in our Next.js project: Shell $ npm install prisma --save-dev To use Prisma, we need to initialize the Prisma Client. Run the following command: Shell $ npx prisma init The above command will create two files: prisma/schema.prisma: The main Prisma configuration file, which will host the database configuration .env: Will contain the database connection URL and other environment variables Configure the Database Used by Prisma We will use SQLite as the database for our Next.js application. Open the schema.prisma file and change the datasource db settings as follows: Shell datasource db { provider = "sqlite" url = env("DATABASE_URL") } Then, in our .env file for the Next.js application, we will change the DATABASE_URL field. Because we’re using SQLite, we need to define the location (which, for SQLite, is a file) where the database will be stored in our application: Shell DATABASE_URL="file:./dev.db" Create a User Model Models represent entities in our app. The model describes how the data should be stored in our database. Prisma takes care of creating tables and fields. Let’s add the following User model in out schema.prisma file: Shell model User { id Int @id @default(autoincrement()) email String @unique name String? flowWalletJobId String? flowWalletAddress String? createdAt DateTime @default(now()) updatedAt DateTime @updatedAt } With our model created, we need to synchronize with the database. For this, Prisma offers a command: Shell $ npx prisma db push Environment variables loaded from .env Prisma schema loaded from prisma/schema.prisma Datasource "db": SQLite database "dev.db" at "file:./dev.db" SQLite database dev.db created at file:./dev.db -> Your database is now in sync with your Prisma schema. Done in 15ms After successfully pushing our users table, we can use Prisma Studio to track our database data. Run the command: Shell $ npx prisma studio Set up the Prisma Client That's it! Our entity and database configuration are complete. Now let's go to the client side. We need to install the Prisma client dependencies in our Next.js app. To do this, run the following command: Shell $ npm install @prisma/client Generate the client from the Prisma schema file: Shell $ npx prisma generate Create a folder named lib in the root folder of your project. Within that folder, create a file entitled prisma.ts. This file will host the client connection. Paste the following code into that file: TypeScript // lib/prisma.ts import { PrismaClient } from '@prisma/client'; let prisma: PrismaClient; if (process.env.NODE_ENV === "production") { prisma = new PrismaClient(); } else { let globalWithPrisma = global as typeof globalThis & { prisma: PrismaClient; }; if (!globalWithPrisma.prisma) { globalWithPrisma.prisma = new PrismaClient(); } prisma = globalWithPrisma.prisma; } export default prisma; Build the Next.js Application Frontend Functionality With our connection on the client part finalized, we can move on to the visual part of our app! Replace the code inside pages/index.tsx file, delete all lines of code and paste in the following code: TypeScript # pages/index.tsx import styles from "@/styles/Home.module.css"; import { Inter } from "next/font/google"; import Head from "next/head"; const inter = Inter({ subsets: ["latin"] }); export default function Home() { return ( <> <Head> <title>Create Next App</title> <meta name="description" content="Generated by create next app" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <link rel="icon" href="/favicon.ico" /> </Head> <main className={styles.main}> <div className={styles.card}> <h1 className={inter.className}>Welcome to Flow Walletless App!</h1> <div style={{ display: "flex", flexDirection: "column", gap: "20px", margin: "20px", } > <button style={{ padding: "20px", width: 'auto' }>Sign Up</button> <button style={{ padding: "20px" }>Sign Out</button> </div> </div> </main> </> ); } In this way, we have the basics and the necessities to illustrate the creation of wallets and accounts! The next step is to configure the Google client to use the Google API to authenticate users. Set up Use of Google OAuth for Authentication We will need Google credentials. For that, open your Google console. Click Create Credentials and select the OAuth Client ID option. Choose Web Application as the application type and define a name for it. We will use the same name: flow_walletless_app. Add http://localhost:3001/api/auth/callback/google as the authorized redirect URI. Click on the Create button. A modal should appear with the Google credentials. We will need the Client ID and Client secret to use in our .env file shortly. Next, we’ll add the next-auth package. To do this, run the following command: Shell $ npm i next-auth Open the .env file and add the following new environment variables to it: Shell GOOGLE_CLIENT_ID= <GOOGLE CLIENT ID> GOOGLE_CLIENT_SECRET=<GOOGLE CLIENT SECRET> NEXTAUTH_URL=http://localhost:3001 NEXTAUTH_SECRET=<YOUR NEXTAUTH SECRET> Paste in your copied Google Client ID and Client Secret. The NextAuth secret can be generated via the terminal with the following command: Shell $ openssl rand -base64 32 Copy the result, which should be a random string of letters, numbers, and symbols. Use this as your value for NEXTAUTH_SECRET in the .env file. Configure NextAuth to Use Google Next.js allows you to create serverless API routes without creating a full backend server. Each file under api is treated like an endpoint. Inside the pages/api/ folder, create a new folder called auth. Then create a file in that folder, called [...nextauth].ts, and add the code below: TypeScript // pages/api/auth/[...nextauth].ts import NextAuth from "next-auth" import GoogleProvider from "next-auth/providers/google"; export default NextAuth({ providers: [ GoogleProvider({ clientId: process.env.GOOGLE_CLIENT_ID as string, clientSecret: process.env.GOOGLE_CLIENT_SECRET as string, }) ], }) Update _app.tsx file to use NextAuth SessionProvider Modify the _app.tsx file found inside the pages folder by adding the SessionProvider from the NextAuth library. Your file should look like this: TypeScript // pages/_app.tsx import "@/styles/globals.css"; import { SessionProvider } from "next-auth/react"; import type { AppProps } from "next/app"; export default function App({ Component, pageProps }: AppProps) { return ( <SessionProvider session={pageProps.session}> <Component {...pageProps} /> </SessionProvider> ); } Update the Main Page To Use NextAuth Functions Let us go back to our index.tsx file in the pages folder. We need to import the functions from the NextAuth library and use them to log users in and out. Our update index.tsx file should look like this: TypeScript // pages/index.tsx import styles from "@/styles/Home.module.css"; import { Inter } from "next/font/google"; import Head from "next/head"; import { useSession, signIn, signOut } from "next-auth/react"; const inter = Inter({ subsets: ["latin"] }); export default function Home() { const { data: session } = useSession(); console.log("session data",session) const signInWithGoogle = () => { signIn(); }; const signOutWithGoogle = () => { signOut(); }; return ( <> <Head> <title>Create Next App</title> <meta name="description" content="Generated by create next app" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <link rel="icon" href="/favicon.ico" /> </Head> <main className={styles.main}> <div className={styles.card}> <h1 className={inter.className}>Welcome to Flow Walletless App!</h1> <div style={{ display: "flex", flexDirection: "column", gap: "20px", margin: "20px", } > <button onClick={signInWithGoogle} style={{ padding: "20px", width: "auto" }>Sign Up</button> <button onClick={signOutWithGoogle} style={{ padding: "20px" }>Sign Out</button> </div> </div> </main> </> ); } Build the “Create User” Endpoint Let us now create a users folder underneath pages/api. Inside this new folder, create a file called index.ts. This file is responsible for: Creating a user (first we check if this user already exists) Calling the Wallet API to create a wallet for this user Calling the Wallet API and retrieving the jobId data if the User entity does not yet have the address created These actions are performed within the handle function, which calls the checkWallet function. Paste the following snippet into your index.ts file: TypeScript // pages/api/users/index.ts import { User } from "@prisma/client"; import { BaseNextRequest, BaseNextResponse } from "next/dist/server/base-http"; import prisma from "../../../lib/prisma"; export default async function handle( req: BaseNextRequest, res: BaseNextResponse ) { const userEmail = JSON.parse(req.body).email; const userName = JSON.parse(req.body).name; try { const user = await prisma.user.findFirst({ where: { email: userEmail, }, }); if (user == null) { await prisma.user.create({ data: { email: userEmail, name: userName, flowWalletAddress: null, flowWalletJobId: null, }, }); } else { await checkWallet(user); } } catch (e) { console.log(e); } } const checkWallet = async (user: User) => { const jobId = user.flowWalletJobId; const address = user.flowWalletAddress; if (address != null) { return; } if (jobId != null) { const request: any = await fetch(`http://localhost:3000/v1/jobs/${jobId}`, { method: "GET", }); const jsonData = await request.json(); if (jsonData.state === "COMPLETE") { const address = await jsonData.result; await prisma.user.update({ where: { id: user.id, }, data: { flowWalletAddress: address, }, }); return; } if (request.data.state === "FAILED") { const request: any = await fetch("http://localhost:3000/v1/accounts", { method: "POST", }); const jsonData = await request.json(); await prisma.user.update({ where: { id: user.id, }, data: { flowWalletJobId: jsonData.jobId, }, }); return; } } if (jobId == null) { const request: any = await fetch("http://localhost:3000/v1/accounts", { method: "POST", }); const jsonData = await request.json(); await prisma.user.update({ where: { id: user.id, }, data: { flowWalletJobId: jsonData.jobId, }, }); return; } }; POST requests to the api/users path will result in calling the handle function. We’ll get to that shortly, but first, we need to create another endpoint for retrieving existing user information. Build the “Get User” Endpoint We’ll create another file in the pages/api/users folder, called getUser.ts. This file is responsible for finding a user in our database based on their email. Copy the following snippet and paste it into getUser.ts: TypeScript // pages/api/users/getUser.ts import prisma from "../../../lib/prisma"; export default async function handle( req: { query: { email: string; }; }, res: any ) { try { const { email } = req.query; const user = await prisma.user.findFirst({ where: { email: email, }, }); return res.json(user); } catch (e) { console.log(e); } } And that's it! With these two files in the pages/api/users folder, we are ready for our Next.js application frontend to make calls to our backend. Add “Create User” and “Get User” Functions to Main Page Now, let’s go back to the pages/index.tsx file to add the new functions that will make the requests to the backend. Replace the contents of index.tsx file with the following snippet: TypeScript // pages/index.tsx import styles from "@/styles/Home.module.css"; import { Inter } from "next/font/google"; import Head from "next/head"; import { useSession, signIn, signOut } from "next-auth/react"; import { useEffect, useState } from "react"; import { User } from "@prisma/client"; const inter = Inter({ subsets: ["latin"] }); export default function Home() { const { data: session } = useSession(); const [user, setUser] = useState<User | null>(null); const signInWithGoogle = () => { signIn(); }; const signOutWithGoogle = () => { signOut(); }; const getUser = async () => { const response = await fetch( `/api/users/getUser?email=${session?.user?.email}`, { method: "GET", } ); const data = await response.json(); setUser(data); return data?.flowWalletAddress != null ? true : false; }; console.log(user) const createUser = async () => { await fetch("/api/users", { method: "POST", body: JSON.stringify({ email: session?.user?.email, name: session?.user?.name }), }); }; useEffect(() => { if (session) { getUser(); createUser(); } }, [session]); return ( <> <Head> <title>Create Next App</title> <meta name="description" content="Generated by create next app" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <link rel="icon" href="/favicon.ico" /> </Head> <main className={styles.main}> <div className={styles.card}> <h1 className={inter.className}>Welcome to Flow Walletless App!</h1> <div style={{ display: "flex", flexDirection: "column", gap: "20px", margin: "20px", } > {user ? ( <div> <h5 className={inter.className}>User Name: {user.name}</h5> <h5 className={inter.className}>User Email: {user.email}</h5> <h5 className={inter.className}>Flow Wallet Address: {user.flowWalletAddress ? user.flowWalletAddress : 'Creating address...'}</h5> </div> ) : ( <button onClick={signInWithGoogle} style={{ padding: "20px", width: "auto" } > Sign Up </button> )} <button onClick={signOutWithGoogle} style={{ padding: "20px" }> Sign Out </button> </div> </div> </main> </> ); } We have added two functions: getUser searches the database for a user with the email logged in. createUser creates a user or updates it if it does not have an address yet. We also added a useEffect that checks if the user is logged in with their Google account. If so, the getUser function is called, returning true if the user exists and has a registered email address. If not, we call the createUser function, which makes the necessary checks and calls. Test Our Next.js Application Finally, we restart our Next.js application with the following command: Shell $ npm run dev You can now sign in with your Google account, and the app will make the necessary calls to our wallet API to create a Flow Testnet address! This is the first step in the walletless Flow process! By following these instructions, your app will create users and accounts in a way that is convenient for the end user. But the wallet API does not stop there. You can do much more with it, such as execute and sign transactions, run scripts to fetch data from the blockchain, and more. Conclusion Account abstraction and walletless onboarding in Flow offer developers a unique solution. By being able to delegate control over accounts, Flow allows developers to create applications that provide users with a seamless onboarding experience. This will hopefully lead to greater adoption of dApps and a new wave of web3 users.

The JVM is an excellent platform for monkey-patching. Monkey patching is a technique used to dynamically update the behavior of a piece of code at run-time. A monkey patch (also spelled monkey-patch, MonkeyPatch) is a way to extend or modify the runtime code of dynamic languages (e.g. Smalltalk, JavaScript, Objective-C, Ruby, Perl, Python, Groovy, etc.) without altering the original source code. — Wikipedia I want to demo several approaches for monkey-patching in Java in this post. As an example, I'll use a sample for-loop. Imagine we have a class and a method. We want to call the method multiple times without doing it explicitly. The Decorator Design Pattern While the Decorator Design Pattern is not monkey-patching, it's an excellent introduction to it anyway. Decorator is a structural pattern described in the foundational book, Design Patterns: Elements of Reusable Object-Oriented Software. The decorator pattern is a design pattern that allows behavior to be added to an individual object, dynamically, without affecting the behavior of other objects from the same class. — Decorator pattern Our use-case is a Logger interface with a dedicated console implementation: We can implement it in Java like this: Java public interface Logger { void log(String message); } public class ConsoleLogger implements Logger { @Override public void log(String message) { System.out.println(message); } } Here's a simple, configurable decorator implementation: Java public class RepeatingDecorator implements Logger { //1 private final Logger logger; //2 private final int times; //3 public RepeatingDecorator(Logger logger, int times) { this.logger = logger; this.times = times; } @Override public void log(String message) { for (int i = 0; i < times; i++) { //4 logger.log(message); } } } Must implement the interface Underlying logger Loop configuration Call the method as many times as necessary Using the decorator is straightforward: Java var logger = new ConsoleLogger(); var threeTimesLogger = new RepeatingDecorator(logger, 3); threeTimesLogger.log("Hello world!"); The Java Proxy The Java Proxy is a generic decorator that allows attaching dynamic behavior: Proxy provides static methods for creating objects that act like instances of interfaces but allow for customized method invocation. — Proxy Javadoc The Spring Framework uses Java Proxies a lot. It's the case of the @Transactional annotation. If you annotate a method, Spring creates a Java Proxy around the encasing class at runtime. When you call it, Spring calls the proxy instead. Depending on the configuration, it opens the transaction or joins an existing one, then calls the actual method, and finally commits (or rollbacks). The API is simple: We can write the following handler: Java public class RepeatingInvocationHandler implements InvocationHandler { private final Logger logger; //1 private final int times; //2 public RepeatingInvocationHandler(Logger logger, int times) { this.logger = logger; this.times = times; } @Override public Object invoke(Object proxy, Method method, Object[] args) throws Exception { if (method.getName().equals("log") && args.length ## 1 && args[0] instanceof String) { //3 for (int i = 0; i < times; i++) { method.invoke(logger, args[0]); //4 } } return null; } } Underlying logger Loop configuration Check every requirement is upheld Call the initial method on the underlying logger Here's how to create the proxy: Java var logger = new ConsoleLogger(); var proxy = (Logger) Proxy.newProxyInstance( //1-2 Main.class.getClassLoader(), new Class[]{Logger.class}, //3 new RepeatingInvocationHandler(logger, 3)); //4 proxy.log("Hello world!"); Create the Proxy object We must cast to Logger as the API was created before generics, and it returns an Object Array of interfaces the object needs to conform to Pass our handler Instrumentation Instrumentation is the capability of the JVM to transform bytecode before it loads it via a Java agent. Two Java agent flavors are available: Static, with the agent passed on the command line when you launch the application Dynamic allows connecting to a running JVM and attaching an agent on it via the Attach API. Note that it represents a huge security issue and has been drastically limited in the latest JDK. The Instrumentation API's surface is limited: As seen above, the API exposes the user to low-level bytecode manipulation via byte arrays. It would be unwieldy to do it directly. Hence, real-life projects rely on bytecode manipulation libraries. ASM has been the traditional library for this, but it seems that Byte Buddy has superseded it. Note that Byte Buddy uses ASM but provides a higher-level abstraction. The Byte Buddy API is outside the scope of this blog post, so let's dive directly into the code: Java public class Repeater { public static void premain(String arguments, Instrumentation instrumentation) { //1 var withRepeatAnnotation = isAnnotatedWith(named("ch.frankel.blog.instrumentation.Repeat")); //2 new AgentBuilder.Default() //3 .type(declaresMethod(withRepeatAnnotation)) //4 .transform((builder, typeDescription, classLoader, module, domain) -> builder //5 .method(withRepeatAnnotation) //6 .intercept( //7 SuperMethodCall.INSTANCE //8 .andThen(SuperMethodCall.INSTANCE) .andThen(SuperMethodCall.INSTANCE)) ).installOn(instrumentation); //3 } } Required signature; it's similar to the main method, with the added Instrumentation argument Match that is annotated with the @Repeat annotation. The DSL reads fluently even if you don't know it (I don't). Byte Buddy provides a builder to create the Java agent Match all types that declare a method with the @Repeat annotation Transform the class accordingly Transform methods annotated with @Repeat Replace the original implementation with the following Call the original implementation three times The next step is to create the Java agent package. A Java agent is a regular JAR with specific manifest attributes. Let's configure Maven to build the agent: XML <plugin> <artifactId>maven-assembly-plugin</artifactId> <!--1--> <configuration> <descriptorRefs> <descriptorRef>jar-with-dependencies</descriptorRef> <!--2--> </descriptorRefs> <archive> <manifestEntries> <Premain-Class>ch.frankel.blog.instrumentation.Repeater</Premain-Class> <!--3--> </manifestEntries> </archive> </configuration> <executions> <execution> <goals> <goal>single</goal> </goals> <phase>package</phase> <!--4--> </execution> </executions> </plugin> Create a JAR containing all dependencies () Testing is more involved, as we need two different codebases, one for the agent and one for the regular code with the annotation. Let's create the agent first: Shell mvn install We can then run the app with the agent: Shell java -javaagent:/Users/nico/.m2/repository/ch/frankel/blog/agent/1.0-SNAPSHOT/agent-1.0-SNAPSHOT-jar-with-dependencies.jar \ #1 -cp ./target/classes #2 ch.frankel.blog.instrumentation.Main #3 Run Java with the agent created in the previous step. The JVM will run the premain method of the class configured in the agent Configure the classpath Set the main class Aspect-Oriented Programming The idea behind AOP is to apply some code across different unrelated object hierarchies - cross-cutting concerns. It's a valuable technique in languages that don't allow traits, code you can graft on third-party objects/classes. Fun fact: I learned about AOP before Proxy. AOP relies on two main concepts: an aspect is the transformation applied to code, while a point cut matches where the aspect applies. In Java, AOP's historical implementation is the excellent AspectJ library. AspectJ provides two approaches, known as weaving: build-time weaving, which transforms the compiled bytecode, and runtime weaving, which relies on the above instrumentation. Either way, AspectJ uses a specific format for aspects and pointcuts. Before Java 5, the format looked like Java but not quite; for example, it used the aspect keyword. With Java 5, one can use annotations in regular Java code to achieve the same goal. We need an AspectJ dependency: XML <dependency> <groupId>org.aspectj</groupId> <artifactId>aspectjrt</artifactId> <version>1.9.19</version> </dependency> As Byte Buddy, AspectJ also uses ASM underneath. Here's the code: Java @Aspect //1 public class RepeatingAspect { @Pointcut("@annotation(repeat) && call(* *(..))") //2 public void callAt(Repeat repeat) {} //3 @Around("callAt(repeat)") //4 public Object around(ProceedingJoinPoint pjp, Repeat repeat) throws Throwable { //5 for (int i = 0; i < repeat.times(); i++) { //6 pjp.proceed(); //7 } return null; } } Mark this class as an aspect Define the pointcut; every call to a method annotated with @Repeat Bind the @Repeat annotation to the the repeat name used in the annotation above Define the aspect applied to the call site; it's an @Around, meaning that we need to call the original method explicitly The signature uses a ProceedingJoinPoint, which references the original method, as well as the @Repeat annotation Loop over as many times as configured Call the original method At this point, we need to weave the aspect. Let's do it at build-time. For this, we can add the AspectJ build plugin: XML <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>aspectj-maven-plugin</artifactId> <executions> <execution> <goals> <goal>compile</goal> <!--1--> </goals> </execution> </executions> </plugin> Bind execution of the plugin to the compile phase To see the demo in effect: Shell mvn compile exec:java -Dexec.mainClass=ch.frankel.blog.aop.Main Java Compiler Plugin Last, it's possible to change the generated bytecode via a Java compiler plugin, introduced in Java 6 as JSR 269. From a bird's eye view, plugins involve hooking into the Java compiler to manipulate the AST in three phases: parse the source code into multiple ASTs, analyze further into Element, and potentially generate source code. The documentation could be less sparse. I found the following Awesome Java Annotation Processing. Here's a simplified class diagram to get you started: I'm too lazy to implement the same as above with such a low-level API. As the expression goes, this is left as an exercise to the reader. If you are interested, I believe the DocLint source code is a good starting point. Conclusion I described several approaches to monkey-patching in Java in this post: the Proxy class, instrumentation via a Java Agent, AOP via AspectJ, and javac compiler plugins. To choose one over the other, consider the following criteria: build-time vs. runtime, complexity, native vs. third-party, and security concerns. To Go Further Monkey patch Guide to Java Instrumentation Byte Buddy Creating a Java Compiler Plugin Awesome Java Annotation Processing Maven AspectJ plugin

This is an article from DZone's 2023 Database Systems Trend Report.For more: Read the Report Good database design is essential to ensure data accuracy, consistency, and integrity and that databases are efficient, reliable, and easy to use. The design must address the storing and retrieving of data quickly and easily while handling large volumes of data in a stable way. An experienced database designer can create a robust, scalable, and secure database architecture that meets the needs of modern data systems. Architecture and Design A modern data architecture for microservices and cloud-native applications involves multiple layers, and each one has its own set of components and preferred technologies. Typically, the foundational layer is constructed as a storage layer, encompassing one or more databases such as SQL, NoSQL, or NewSQL. This layer assumes responsibility for the storage, retrieval, and management of data, including tasks like indexing, querying, and transaction management. To enhance this architecture, it is advantageous to design a data access layer that resides above the storage layer but below the service layer. This data access layer leverages technologies like object-relational mapping or data access objects to simplify data retrieval and manipulation. Finally, at the topmost layer lies the presentation layer, where the information is skillfully presented to the end user. The effective transmission of data through the layers of an application, culminating in its presentation as meaningful information to users, is of utmost importance in a modern data architecture. The goal here is to design a scalable database with the ability to handle a high volume of traffic and data while minimizing downtime and performance issues. By following best practices and addressing a few challenges, we can meet the needs of today's modern data architecture for different applications. Figure 1: Layered architecture Considerations By taking into account the following considerations when designing a database for enterprise-level usage, it is possible to create a robust and efficient system that meets the specific needs of the organization while ensuring data integrity, availability, security, and scalability. One important consideration is the data that will be stored in the database. This involves assessing the format, size, complexity, and relationships between data entities. Different types of data may require specific storage structures and data models. For instance, transactional data often fits well with a relational database model, while unstructured data like images or videos may require a NoSQL database model. The frequency of data retrieval or access plays a significant role in determining the design considerations. In read-heavy systems, implementing a cache for frequently accessed data can enhance query response times. Conversely, the emphasis may be on lower data retrieval frequencies for data warehouse scenarios. Techniques such as indexing, caching, and partitioning can be employed to optimize query performance. Ensuring the availability of the database is crucial for maintaining optimal application performance. Techniques such as replication, load balancing, and failover are commonly used to achieve high availability. Additionally, having a robust disaster recovery plan in place adds an extra layer of protection to the overall database system. As data volumes grow, it is essential that the database system can handle increased loads without compromising performance. Employing techniques like partitioning, sharding, and clustering allows for effective scalability within a database system. These approaches enable the efficient distribution of data and workload across multiple servers or nodes. Data security is a critical consideration in modern database design, given the rising prevalence of fraud and data breaches. Implementing robust access controls, encryption mechanisms for sensitive personally identifiable information, and conducting regular audits are vital for enhancing the security of a database system. In transaction-heavy systems, maintaining consistency in transactional data is paramount. Many databases provide features such as appropriate locking mechanisms and transaction isolation levels to ensure data integrity and consistency. These features help to prevent issues like concurrent data modifications and inconsistencies. Challenges Determining the most suitable tool or technology for our database needs can be a challenge due to the rapid growth and evolving nature of the database landscape. With different types of databases emerging daily and even variations among vendors offering the same type, it is crucial to plan carefully based on your specific use cases and requirements. By thoroughly understanding our needs and researching the available options, we can identify the right tool with the appropriate features to meet our database needs effectively. Polyglot persistence is a consideration that arises from the demand of certain applications, leading to the use of multiple SQL or NoSQL databases. Selecting the right databases for transactional systems, ensuring data consistency, handling financial data, and accommodating high data volumes pose challenges. Careful consideration is necessary to choose the appropriate databases that can fulfill the specific requirements of each aspect while maintaining overall system integrity. Integrating data from different upstream systems, each with its own structure and volume, presents a significant challenge. The goal is to achieve a single source of truth by harmonizing and integrating the data effectively. This process requires comprehensive planning to ensure compatibility and future-proofing the integration solution to accommodate potential changes and updates. Performance is an ongoing concern in both applications and database systems. Every addition to the database system can potentially impact performance. To address performance issues, it is essential to follow best practices when adding, managing, and purging data, as well as properly indexing, partitioning, and implementing encryption techniques. By employing these practices, you can mitigate performance bottlenecks and optimize the overall performance of your database system. Considering these factors will contribute to making informed decisions and designing an efficient and effective database system for your specific requirements. Advice for Building Your Architecture Goals for a better database design should include efficiency, scalability, security, and compliance. In the table below, each goal is accompanied by its corresponding industry expectation, highlighting the key aspects that should be considered when designing a database for optimal performance, scalability, security, and compliance. GOALS FOR DATABASE DESIGN Goal Industry Expectation Efficiency Optimal performance and responsiveness of the database system, minimizing latency and maximizing throughput. Efficient handling of data operations, queries, and transactions. Scalability Ability to handle increasing data volumes, user loads, and concurrent transactions without sacrificing performance. Scalable architecture that allows for horizontal or vertical scaling to accommodate growth. Security Robust security measures to protect against unauthorized access, data breaches, and other security threats. Implementation of access controls, encryption, auditing mechanisms, and adherence to industry best practices and compliance regulations. Compliance Adherence to relevant industry regulations, standards, and legal requirements. Ensuring data privacy, confidentiality, and integrity. Implementing data governance practices and maintaining audit trails to demonstrate compliance. Table 1 When building your database architecture, it's important to consider several key factors to ensure the design is effective and meets your specific needs. Start by clearly defining the system's purpose, data types, volume, access patterns, and performance expectations. Consider clear requirements that provide clarity on the data to be stored and the relationships between the data entities. This will help ensure that the database design aligns with quality standards and conforms to your requirements. Also consider normalization, which enables efficient storage use by minimizing redundant data, improves data integrity by enforcing consistency and reliability, and facilitates easier maintenance and updates. Selecting the right database model or opting for polyglot persistence support is crucial to ensure the database aligns with your specific needs. This decision should be based on the requirements of your application and the data it handles. Planning for future growth is essential to accommodate increasing demand. Consider scalability options that allow your database to handle growing data volumes and user loads without sacrificing performance. Alongside growth, prioritize data protection by implementing industry-standard security recommendations and ensuring appropriate access levels are in place and encourage implementing IT security measures to protect the database from unauthorized access, data theft, and security threats. A good back-up system is a testament to the efficiency of a well-designed database. Regular backups and data synchronization, both on-site and off-site, provide protection against data loss or corruption, safeguarding your valuable information. To validate the effectiveness of your database design, test the model using sample data from real-world scenarios. This testing process will help validate the performance, reliability, and functionality of the database system you are using, ensuring it meets your expectations. Good documentation practices play a vital role in improving feedback systems and validating thought processes and implementations during the design and review phases. Continuously improving documentation will aid in future maintenance, troubleshooting, and system enhancement efforts. Primary and secondary keys contribute to data integrity and consistency. Use indexes to optimize database performance by indexing frequently queried fields and limiting the number of fields returned in queries. Regularly backing up the database protects against data loss during corruption, system failure, or other unforeseen circumstances. Data archiving and purging practices help remove infrequently accessed data, reducing the size of the active dataset. Proper error handling and logging aid in debugging, troubleshooting, and system maintenance. Regular maintenance is crucial for growing database systems. Plan and schedule regular backups, perform performance tuning, and stay up to date with software upgrades to ensure optimal database performance and stability. Conclusion Designing a modern data architecture that can handle the growing demands of today's digital world is not an easy job. However, if you follow best practices and take advantage of the latest technologies and techniques, it is very much possible to build a scalable, flexible, and secure database. It just requires the right mindset and your commitment to learning and improving with a proper feedback loop. Additional reading: Semantic Modeling for Data: Avoiding Pitfalls and Breaking Dilemmas by Panos Alexopoulos Learn PostgreSQL: Build and manage high-performance database solutions using PostgreSQL 12 and 13 by Luca Ferrari and Enrico Pirozzi Designing Data-Intensive Applications by Martin Kleppmann This is an article from DZone's 2023 Database Systems Trend Report.For more: Read the Report

In this article, we delve into the exciting realm of containerizing Helidon applications, followed by deploying them effortlessly to a Kubernetes environment. To achieve this, we'll harness the power of JKube’s Kubernetes Maven Plugin, a versatile tool for Java applications for Kubernetes deployments that has recently been updated to version 1.14.0. What's exciting about this release is that it now supports the Helidon framework, a Java Microservices gem open-sourced by Oracle in 2018. If you're curious about Helidon, we've got some blog posts to get you up to speed: Building Microservices With Oracle Helidon Ultra-Fast Microservices: When MicroStream Meets Helidon Helidon: 2x Productivity With Microprofile REST Client In this article, we will closely examine the integration between JKube’s Kubernetes Maven Plugin and Helidon. Here's a sneak peek of the exciting journey we'll embark on: We'll kick things off by generating a Maven application from Helidon Starter Transform your Helidon application into a nifty Docker image. Craft Kubernetes YAML manifests tailored for your Helidon application. Apply those manifests to your Kubernetes cluster. We'll bundle those Kubernetes YAML manifests into a Helm Chart. We'll top it off by pushing that Helm Chart to a Helm registry. Finally, we'll deploy our Helidon application to Red Hat OpenShift. An exciting aspect worth noting is that JKube’s Kubernetes Maven Plugin can be employed with previous versions of Helidon projects as well. The only requirement is to provide your custom image configuration. With this latest release, Helidon users can now easily generate opinionated container images. Furthermore, the plugin intelligently detects project dependencies and seamlessly incorporates Kubernetes health checks into the generated manifests, streamlining the deployment process. Setting up the Project You can either use an existing Helidon project or create a new one from Helidon Starter. If you’re on JDK 17 use 3.x version of Helidon. Otherwise, you can stick to Helidon 2.6.x which works with older versions of Java. In the starter form, you can choose either Helidon SE or Helidon Microprofile, choose application type, and fill out basic details like project groupId, version, and artifactId. Once you’ve set your project, you can add JKube’s Kubernetes Maven Plugin to your pom.xml: XML <plugin> <groupId>org.eclipse.jkube</groupId> <artifactId>kubernetes-maven-plugin</artifactId> <version>1.14.0</version> </plugin> Also, the plugin version is set to 1.14.0, which is the latest version at the time of writing. You can check for the latest version on the Eclipse JKube releases page. It’s not really required to add the plugin if you want to execute it directly from some CI pipeline. You can just provide a fully qualified name of JKube’s Kubernetes Maven Plugin while issuing some goals like this: Shell $ mvn org.eclipse.jkube:kubernetes-maven-plugin:1.14.0:resource Now that we’ve added the plugin to the project, we can start using it. Creating Container Image (JVM Mode) In order to build a container image, you do not need to provide any sort of configuration. First, you need to build your project. Shell $ mvn clean install Then, you just need to run k8s:build goal of JKube’s Kubernetes Maven Plugin. By default, it builds the image using the Docker build strategy, which requires access to a Docker daemon. If you have access to a docker daemon, run this command: Shell $ mvn k8s:build If you don’t have access to any docker daemon, you can also build the image using the Jib build strategy: Shell $ mvn k8s:build -Djkube.build.strategy=jib You will notice that Eclipse JKube has created an opinionated container image for your application based on your project configuration. Here are some key points about JKube’s Kubernetes Maven Plugin to observe in this zero configuration mode: It used quay.io/jkube/jkube-java as a base image for the container image It added some labels to the container image (picked from pom.xml) It exposed some ports in the container image based on the project configuration It automatically copied relevant artifacts and libraries required to execute the jar in the container environment. Creating Container Image (Native Mode) In order to create a container image for the native executable, we need to generate the native executable first. In order to do that, let’s build our project in the native-image profile (as specified in Helidon GraalVM Native Image documentation): Shell $ mvn package -Pnative-image This creates a native executable file in the target folder of your project. In order to create a container image based on this executable, we just need to run k8s:build goal but also specify native-image profile: Shell $ mvn k8s:build -Pnative-image Like JVM mode, Eclipse JKube creates an opinionated container image but uses a lightweight base image: registry.access.redhat.com/ubi8/ubi-minimal and exposes only the required ports by application. Customizing Container Image as per Requirements Creating a container image with no configuration is a really nice way to get started. However, it might not suit everyone’s use case. Let’s take a look at how to configure various aspects of the generated container image. You can override basic aspects of the container image with some properties like this: Property Name Description jkube.generator.name Change Image Name jkube.generator.from Change Base Image jkube.generator.tags A comma-separated value of additional tags for the image If you want more control, you can provide a complete XML configuration for the image in the plugin configuration section: XML <plugin> <groupId>org.eclipse.jkube</groupId> <artifactId>kubernetes-maven-plugin</artifactId> <version>${jkube.version}</version> <configuration> <images> <image> <name>${project.artifactId}:${project.version}</name> <build> <from>openjdk:11-jre-slim</from> <ports>8080</ports> <assembly> <mode>dir</mode> <targetDir>/deployments</targetDir> <layers> <layer> <id>lib</id> <fileSets> <fileSet> <directory>${project.basedir}/target/libs</directory> <outputDirectory>libs</outputDirectory> <fileMode>0640</fileMode> </fileSet> </fileSets> </layer> <layer> <id>app</id> <files> <file> <source>${project.basedir}/target/${project.artifactId}.jar</source> <outputDirectory>.</outputDirectory> </file> </files> </layer> </layers> </assembly> <cmd>java -jar /deployments/${project.artifactId}.jar</cmd> </build> </image> </images> </configuration> </plugin> The same is also possible by providing your own Dockerfile in the project base directory. Kubernetes Maven Plugin automatically detects it and builds a container image based on its content: Dockerfile FROM openjdk:11-jre-slim COPY maven/target/helidon-quickstart-se.jar /deployments/ COPY maven/target/libs /deployments/libs CMD ["java", "-jar", "/deployments/helidon-quickstart-se.jar"] EXPOSE 8080 Pushing the Container Image to Quay.io: Once you’ve built a container image, you most likely want to push it to some public or private container registry. Before pushing the image, make sure you’ve renamed your image to include the registry name and registry user. If I want to push an image to Quay.io in the namespace of a user named rokumar, this is how I would need to rename my image: Shell $ mvn k8s:build -Djkube.generator.name=quay.io/rokumar/%a:%v %a and %v correspond to project artifactId and project version. For more information, you can check the Kubernetes Maven Plugin Image Configuration documentation. Once we’ve built an image with the correct name, the next step is to provide credentials for our registry to JKube’s Kubernetes Maven Plugin. We can provide registry credentials via the following sources: Docker login Local Maven Settings file (~/.m2/settings.xml) Provide it inline using jkube.docker.username and jkube.docker.password properties Once you’ve configured your registry credentials, you can issue the k8s:push goal to push the image to your specified registry: Shell $ mvn k8s:push Generating Kubernetes Manifests In order to generate opinionated Kubernetes manifests, you can use k8s:resource goal from JKube’s Kubernetes Maven Plugin: Shell $ mvn k8s:resource It generates Kubernetes YAML manifests in the target directory: Shell $ ls target/classes/META-INF/jkube/kubernetes helidon-quickstart-se-deployment.yml helidon-quickstart-se-service.yml JKube’s Kubernetes Maven Plugin automatically detects if the project contains io.helidon:helidon-health dependency and adds liveness, readiness, and startup probes: YAML $ cat target/classes/META-INF/jkube/kubernetes//helidon-quickstart-se-deployment. yml | grep -A8 Probe livenessProbe: failureThreshold: 3 httpGet: path: /health/live port: 8080 scheme: HTTP initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 -- readinessProbe: failureThreshold: 3 httpGet: path: /health/ready port: 8080 scheme: HTTP initialDelaySeconds: 0 periodSeconds: 10 successThreshold: 1 Applying Kubernetes Manifests JKube’s Kubernetes Maven Plugin provides k8s:apply goal that is equivalent to kubectl apply command. It just applies the resources generated by k8s:resource in the previous step. Shell $ mvn k8s:apply Packaging Helm Charts Helm has established itself as the de facto package manager for Kubernetes. You can package generated manifests into a Helm Chart and apply it on some other cluster using Helm CLI. You can generate a Helm Chart of generated manifests using k8s:helm goal. The interesting thing is that JKube’s Kubernetes Maven Plugin doesn’t rely on Helm CLI for generating the chart. Shell $ mvn k8s:helm You’d notice Helm Chart is generated in target/jkube/helm/ directory: Shell $ ls target/jkube/helm/helidon-quickstart-se/kubernetes Chart.yaml helidon-quickstart-se-0.0.1-SNAPSHOT.tar.gz README.md templates values.yaml Pushing Helm Charts to Helm Registries Usually, after generating a Helm Chart locally, you would want to push it to some Helm registry. JKube’s Kubernetes Maven Plugin provides k8s:helm-push goal for achieving this task. But first, we need to provide registry details in plugin configuration: XML <plugin> <groupId>org.eclipse.jkube</groupId> <artifactId>kubernetes-maven-plugin</artifactId> <version>1.14.0</version> <configuration> <helm> <snapshotRepository> <name>ChartMuseum</name> <url>http://example.com/api/charts</url> <type>CHARTMUSEUM</type> <username>user1</username> </snapshotRepository> </helm> </configuration> </plugin> JKube’s Kubernetes Maven Plugin supports pushing Helm Charts to ChartMuseum, Nexus, Artifactory, and OCI registries. You have to provide the applicable Helm repository type and URL. You can provide the credentials via environment variables, properties, or ~/.m2/settings.xml. Once you’ve all set up, you can run k8s:helm-push goal to push chart: Shell $ mvn k8s:helm-push -Djkube.helm.snapshotRepository.password=yourpassword Deploying To Red Hat OpenShift If you’re deploying to Red Hat OpenShift, you can use JKube’s OpenShift Maven Plugin to deploy your Helidon application to an OpenShift cluster. It contains some add-ons specific to OpenShift like S2I build strategy, support for Routes, etc. You also need to add the JKube’s OpenShift Maven Plugin plugin to your pom.xml. Maybe you can add it in a separate profile: XML <profile> <id>openshift</id> <build> <plugins> <plugin> <groupId>org.eclipse.jkube</groupId> <artifactId>openshift-maven-plugin</artifactId> <version>${jkube.version}</version> </plugin> </plugins> </build> </profile> Then, you can deploy the application with a combination of these goals: Shell $ mvn oc:build oc:resource oc:apply -Popenshift Conclusion In this article, you learned how smoothly you can deploy your Helidon applications to Kubernetes using Eclipse JKube’s Kubernetes Maven Plugin. We saw how effortless it is to package your Helidon application into a container image and publish it to some container image registry. We can alternatively generate Helm Charts of our Kubernetes YAML manifests and publish Helm Charts to some Helm registry. In the end, we learned about JKube’s OpenShift Maven Plugin, which is specifically designed for Red Hat OpenShift users who want to deploy their Helidon applications to Red Hat OpenShift. You can find the code used in this blog post in this GitHub repository. In case you’re interested in knowing more about Eclipse JKube, you can check these links: Documentation Github Issue Tracker StackOverflow YouTube Channel Twitter Gitter Chat

Agile estimation plays a pivotal role in Agile project management, enabling teams to gauge the effort, time, and resources necessary to accomplish their tasks. Precise estimations empower teams to efficiently plan their work, manage expectations, and make well-informed decisions throughout the project's duration. In this article, we delve into various Agile estimation techniques and best practices that enhance the accuracy of your predictions and pave the way for your team's success. The Essence of Agile Estimation Agile estimation is an ongoing, iterative process that takes place at different levels of detail, ranging from high-level release planning to meticulous sprint planning. The primary objective of Agile estimation is to provide just enough information for teams to make informed decisions without expending excessive time on analysis and documentation. Designed to be lightweight, collaborative, and adaptable, Agile estimation techniques enable teams to rapidly adjust their plans as new information emerges or priorities shift. Prominent Agile Estimation Techniques 1. Planning Poker Planning Poker is a consensus-driven estimation technique that employs a set of cards with pre-defined numerical values, often based on the Fibonacci sequence (1, 2, 3, 5, 8, 13, etc.). Each team member selects a card representing their estimate for a specific task, and all cards are revealed simultaneously. If there is a significant discrepancy in estimates, team members deliberate their reasoning and repeat the process until a consensus is achieved. 2. T-Shirt Sizing T-shirt sizing is a relative estimation technique that classifies tasks into different "sizes" according to their perceived complexity or effort, such as XS, S, M, L, and XL. This method allows teams to swiftly compare tasks and prioritize them based on their relative size. Once tasks are categorized, more precise estimation techniques can be employed if needed. 3. User Story Points User story points serve as a unit of measurement to estimate the relative effort required to complete a user story. This technique entails assigning a point value to each user story based on its complexity, risk, and effort, taking into account factors such as workload, uncertainty, and potential dependencies. Teams can then use these point values to predict the number of user stories they can finish within a given timeframe. 4. Affinity Estimation Affinity Estimation is a technique that involves grouping tasks or user stories based on their similarities in terms of effort, complexity, and size. This method helps teams quickly identify patterns and relationships among tasks, enabling them to estimate more efficiently. Once tasks are grouped, they can be assigned a relative point value or size category. 5. Wideband Delphi The Wideband Delphi method is a consensus-based estimation technique that involves multiple rounds of anonymous estimation and feedback. Team members individually provide estimates for each task, and then the estimates are shared anonymously with the entire team. Team members discuss the range of estimates and any discrepancies before submitting revised estimates in subsequent rounds. This process continues until a consensus is reached. Risk Management in Agile Estimation Identify and Assess Risks Incorporate risk identification and assessment into your Agile estimation process. Encourage team members to consider potential risks associated with each task or user story, such as technical challenges, dependencies, or resource constraints. By identifying and assessing risks early on, your team can develop strategies to mitigate them, leading to more accurate estimates and a smoother project execution. Assign Risk Factors Assign risk factors to tasks or user stories based on their level of uncertainty or potential impact on the project. These risk factors can be numerical values or qualitative categories (e.g., low, medium, high) that help your team prioritize tasks and allocate resources effectively. Incorporating risk factors into your estimates can provide a more comprehensive understanding of the work involved and help your team make better-informed decisions. Risk-Based Buffering Include risk-based buffering in your Agile estimation process by adding contingency buffers to account for uncertainties and potential risks. These buffers can be expressed as additional time, resources, or user story points, and they serve as a safety net to ensure that your team can adapt to unforeseen challenges without jeopardizing the project's success. Monitor and Control Risks Continuously monitor and control risks throughout the project lifecycle by regularly reviewing your risk assessments and updating them as new information becomes available. This proactive approach allows your team to identify emerging risks and adjust their plans accordingly, ensuring that your estimates remain accurate and relevant. Learn From Risks Encourage your team to learn from the risks encountered during the project and use this knowledge to improve their estimation and risk management practices. Conduct retrospective sessions to discuss the risks faced, their impact on the project, and the effectiveness of the mitigation strategies employed. By learning from past experiences, your team can refine its risk management approach and enhance the accuracy of future estimates. By incorporating risk management into your Agile estimation process, you can help your team better anticipate and address potential challenges, leading to more accurate estimates and a higher likelihood of project success. This approach also fosters a culture of proactive risk management and continuous learning within your team, further enhancing its overall effectiveness and adaptability. Best Practices for Agile Estimation Foster Team Collaboration Efficient Agile estimation necessitates input from all team members, as each individual contributes unique insights and perspectives. Promote open communication and collaboration during estimation sessions to ensure everyone's opinions are considered and to cultivate a shared understanding of the tasks at hand. Utilize Historical Data Draw upon historical data from previous projects or sprints to inform your estimations. Examining past performance can help teams identify trends, patterns, and areas for improvement, ultimately leading to more accurate predictions in the future. Velocity and Capacity Planning Incorporate team velocity and capacity planning into your Agile estimation process. Velocity is a measure of the amount of work a team can complete within a given sprint or iteration, while capacity refers to the maximum amount of work a team can handle. By considering these factors, you can ensure that your estimates align with your team's capabilities and avoid overcommitting to work. Break Down Large Tasks Large tasks or user stories can be challenging to estimate accurately. Breaking them down into smaller, more manageable components can make the estimation process more precise and efficient. Additionally, this approach helps teams better understand the scope and complexity of the work involved, leading to more realistic expectations and improved planning. Revisit Estimates Regularly Agile estimation is a continuous process, and teams should be prepared to revise their estimates as new information becomes available or circumstances change. Periodically review and update your estimates to ensure they remain accurate and pertinent throughout the project lifecycle. Acknowledge Uncertainty Agile estimation recognizes the inherent uncertainty in software development. Instead of striving for flawless predictions, focus on providing just enough information to make informed decisions and be prepared to adapt as necessary. Establish a Baseline Create a baseline for your estimates by selecting a well-understood task or user story as a reference point. This baseline can help teams calibrate their estimates and ensure consistency across different tasks and projects. Pursue Continuous Improvement Consider Agile estimation as an opportunity for ongoing improvement. Reflect on your team's estimation accuracy and pinpoint areas for growth. Experiment with different techniques and practices to discover what works best for your team and refine your approach over time. Conclusion Agile estimation is a vital component of successful Agile project management. By employing the appropriate techniques and adhering to best practices, teams can enhance their ability to predict project scope, effort, and duration, resulting in more effective planning and decision-making. Keep in mind that Agile estimation is an iterative process, and teams should continuously strive to learn from their experiences and refine their approach for even greater precision in the future.

Sorting is a fundamental operation in computer science and is crucial for organizing and processing large sets of data efficiently. There are numerous sorting algorithms available, each with its unique characteristics and trade-offs. Whether you’re a beginner programmer or an experienced developer, understanding sorting algorithms is essential for optimizing your code and solving real-world problems efficiently. Sorting algorithms play a crucial role in computer science and programming, enabling efficient organization and retrieval of data. In this article, we will dive into the world of sorting algorithms, exploring their various types, their strengths, and their best use cases. Understanding these algorithms will empower you to choose the most suitable sorting technique for your specific requirements. What Are Sorting Algorithms? Sorting algorithms are algorithms designed to arrange elements in a specific order, typically ascending or descending. They are fundamental tools in computer science and play a vital role in data organization and retrieval. Sorting algorithms take an unsorted collection of elements and rearrange them according to a predetermined criterion, allowing for easier searching, filtering, and analysis of data. The primary goal of sorting algorithms is to transform a disordered set of elements into a sequence that follows a specific order. The order can be based on various factors, such as numerical value, alphabetical order, or custom-defined criteria. Sorting algorithms operate on different data structures, including arrays, lists, trees, and more. These algorithms come in various types, each with its own set of characteristics, efficiency, and suitability for different scenarios. Some sorting algorithms are simple and easy to implement, while others are more complex but offer improved performance for larger datasets. The choice of sorting algorithm depends on factors such as the size of the dataset, the expected order of the input, stability requirements, memory constraints, and desired time complexity. Sorting algorithms are not limited to a specific programming language or domain. They are widely used in a range of applications, including databases, search algorithms, data analysis, graph algorithms, and more. Understanding sorting algorithms is essential for developers and computer scientists, as it provides the foundation for efficient data manipulation and retrieval. Types of Sorting Algorithms Bubble Sort Bubble Sort is a simple and intuitive algorithm that repeatedly swaps adjacent elements if they are in the wrong order. It continues this process until the entire list is sorted. While easy to understand and implement, Bubble Sort has a time complexity of O(n²) in the worst case, making it inefficient for large datasets. It is primarily useful for educational purposes or when dealing with small datasets. Insertion Sort Insertion Sort works by dividing the list into a sorted and an unsorted part. It iterates through the unsorted part, comparing each element to the elements in the sorted part and inserting it at the correct position. Insertion Sort has a time complexity of O(n²) in the worst case but performs better than Bubble Sort in practice, particularly for partially sorted or small datasets. Selection Sort Selection Sort divides the list into a sorted and an unsorted part, similar to Insertion Sort. However, instead of inserting elements, it repeatedly finds the minimum element from the unsorted part and swaps it with the first element of the unsorted part. Selection Sort has a time complexity of O(n²) and is generally less efficient than Insertion Sort or more advanced algorithms. It is mainly used for educational purposes or small datasets. Merge Sort Merge Sort is a divide-and-conquer algorithm that recursively divides the list into smaller halves, sorts them, and then merges them back together. It has a time complexity of O(n log n), making it more efficient than the previous algorithms for large datasets. Merge Sort is known for its stability (preserving the order of equal elements) and is widely used in practice. Quick Sort Quick Sort, another divide-and-conquer algorithm, selects a “pivot” element and partitions the list around it such that all elements less than the pivot come before it, and all elements greater come after it. The algorithm then recursively sorts the two partitions. Quick Sort has an average time complexity of O(n log n), but it can degrade to O(n²) in the worst case. However, its efficient average-case performance and in-place sorting make it a popular choice for sorting large datasets. Heap Sort Heap Sort uses a binary heap data structure to sort the elements. It first builds a heap from the input list, then repeatedly extracts the maximum element (root) and places it at the end of the sorted portion. Heap Sort has a time complexity of O(n log n) and is often used when a guaranteed worst-case performance is required. Radix Sort Radix Sort is a non-comparative algorithm that sorts elements by processing individual digits or bits of the elements. It works by grouping elements based on each digit’s value and repeatedly sorting them until the entire list is sorted. Radix Sort has a time complexity of O(k * n), where k is the number of digits or bits in the input elements. It is particularly efficient for sorting integers or fixed-length strings. Choosing the Right Sorting Algorithm Choosing the right sorting algorithm depends on several factors, including the characteristics of the data set, the desired order, time complexity requirements, stability considerations, and memory constraints. Here are some key considerations to help you make an informed decision: Input Size: Consider the size of your data set. Some sorting algorithms perform better with smaller data sets, while others excel with larger inputs. For small data sets, simple algorithms like Bubble Sort or Insertion Sort may be sufficient. However, for larger data sets, more efficient algorithms like Merge Sort, Quick Sort, or Heap Sort are generally preferred due to their lower time complexity. Input Order: Take into account the initial order of the data set. If the data is already partially sorted or nearly sorted, algorithms like Insertion Sort or Bubble Sort can be advantageous as they have better performance under these conditions. They tend to have a lower time complexity when dealing with partially ordered inputs. Stability: Consider whether the stability of the sorting algorithm is important for your use case. A stable sorting algorithm preserves the relative order of elements with equal keys. If maintaining the original order of equal elements is crucial, algorithms like Merge Sort or Insertion Sort are stable options, while Quick Sort is not inherently stable. Time Complexity: Analyze the time complexity requirements for your application. Different sorting algorithms have varying time complexities. For example, Bubble Sort and Insertion Sort have average and worst-case time complexities of O(n²), making them less efficient for large data sets. Merge Sort and Heap Sort have average and worst-case time complexities of O(n log n), offering better performance for larger data sets. Quick Sort has an average time complexity of O(n log n), but its worst-case time complexity can reach O(n²) in certain scenarios. Memory Usage: Consider the memory requirements of the sorting algorithm. In-place algorithms modify the original data structure without requiring significant additional memory. Algorithms like Insertion Sort, Quick Sort, and Heap Sort can be implemented in place, which is beneficial when memory usage is a concern. On the other hand, algorithms like Merge Sort require additional memory proportional to the input size, as they create temporary arrays during the merging process. Specialized Requirements: Depending on the specific characteristics of your data or the desired order, there may be specialized sorting algorithms that offer advantages. For example, Radix Sort is useful for sorting integers or strings based on individual digits or characters. Conclusion In computer science and programming, sorting algorithms are essential for the effective manipulation and analysis of data. Although some of the most popular sorting algorithms were described in this article, it's crucial to remember that there are a variety of other variations and specialized algorithms that are also accessible. The sorting algorithm to use depends on a number of variables, including the dataset's size, distribution, memory requirements, and desired level of time complexity. Making educated selections and optimizing your code for particular contexts requires an awareness of the fundamentals and traits of various sorting algorithms. Overall, sorting algorithms are powerful tools that enable efficient organization and retrieval of data. They allow us to transform unordered collections into ordered sequences, facilitating faster and easier data processing in various computational tasks.

I remember the first time I saw a demonstration of Ruby on Rails. With very little effort, demonstrators created a full-stack web application that could be used for real business purposes. I was impressed – especially when I thought about how much time it took me to deliver similar solutions using the Seam and Struts frameworks. Ruby was created in 1993 to be an easy-to-use scripting language that also included object-oriented features. Ruby on Rails took things to the next level in the mid 2000s – arriving at the right time to become the tech-of-choice for the initial startup efforts of Twitter, Shopify, GitHub, and Airbnb. I began to ask the question, “Is it possible to have a product, like Ruby on Rails, without needing to worry about the infrastructure or underlying data tier?” That’s when I discovered the Zipper platform. About Zipper Zipper is a platform for building web services using simple TypeScript functions. You use Zipper to create applets (not related to Java, though they share the same name), which are then built and deployed on Zipper’s platform. The coolest thing about Zipper is that it lets you focus on coding your solution using TypeScript, and you don’t need to worry about anything else. Zipper takes care of: User interface Infrastructure to host your solution Persistence layer APIs to interact with your applet Authentication Although the platform is currently in beta, it’s open for consumers to use. At the time I wrote this article, there were four templates in place to help new adopters get started: Hello World – a basic applet to get you started CRUD Template – offers a ToDo list where items can be created, viewed, updated, and deleted Slack App Template – provides an example on how to interact with the Slack service AI-Generated Code – expresses your solution in human language and lets AI create an applet for you There is also a gallery on the Zipper platform that provides applets that can be forked in the same manner as Git-based repositories. I thought I would put the Zipper platform to the test and create a ballot applet. HOA Ballot Use Case The homeowner’s association (HOA) concept started to gain momentum in the United States back in the 20th century. Subdivisions formed HOAs to handle things like the care of common areas and for establishing rules and guidelines for residents. Their goal is to maintain the subdivision’s quality of living as a whole, long after the home builder has finished development. HOAs often hold elections to allow homeowners to vote on the candidate they feel best matches their views and perspectives. In fact, last year I published an article on how an HOA ballot could be created using Web3 technologies. For this article, I wanted to take the same approach using Zipper. Ballot Requirements The requirements for the ballot applet are: As a ballot owner, I need the ability to create a list of candidates for the ballot. As a ballot owner, I need the ability to create a list of registered voters. As a voter, I need the ability to view the list of candidates. As a voter, I need the ability to cast one vote for a single candidate. As a voter, I need the ability to see a current tally of votes that have been cast for each candidate. Additionally, I thought some stretch goals would be nice too: As a ballot owner, I need the ability to clear all candidates. As a ballot owner, I need the ability to clear all voters. As a ballot owner, I need the ability to set a title for the ballot. As a ballot owner, I need the ability to set a subtitle for the ballot. Designing the Ballot Applet To start working on the Zipper platform, I navigated to Zipper's website and clicked the Sign In button. Next, I selected an authentication source: Once logged in, I used the Create Applet button from the dashboard to create a new applet: A unique name is generated, but that can be changed to better identify your use case. For now, I left all the defaults the same and pushed the Next button – which allowed me to select from four different templates for applet creation. I started with the CRUD template because it provides a solid example of how the common create, view, update, and delete flows work on the Zipper platform. Once the code was created, the screen appears as shown below: With a fully functional applet in place, we can now update the code to meet the HOA ballot use requirements. Establish Core Elements For the ballot applet, the first thing I wanted to do was update the types.ts file as shown below: TypeScript export type Candidate = { id: string; name: string; votes: number; }; export type Voter = { email: string; name: string; voted: boolean; }; I wanted to establish constant values for the ballot title and subtitle within a new file called constants.ts: TypeScript export class Constants { static readonly BALLOT_TITLE = "Sample Ballot"; static readonly BALLOT_SUBTITLE = "Sample Ballot Subtitle"; }; To allow only the ballot owner to make changes to the ballot, I used the Secrets tab for the applet to create an owner secret with the value of my email address. Then I introduced a common.ts file which contained the validateRequest() function: TypeScript export function validateRequest(context: Zipper.HandlerContext) { if (context.userInfo?.email !== Deno.env.get('owner')) { return ( <> <Markdown> {`### Error: You are not authorized to perform this action`} </Markdown> </> ); } }; This way I could pass in the context to this function to make sure only the value in the owner secret would be allowed to make changes to the ballot and voters. Establishing Candidates After understanding how the ToDo item was created in the original CRUD applet, I was able to introduce the create-candidate.ts file as shown below: TypeScript import { Candidate } from "./types.ts"; import { validateRequest } from "./common.ts"; type Input = { name: string; }; export async function handler({ name }: Input, context: Zipper.HandlerContext) { validateRequest(context); const candidates = (await Zipper.storage.get<Candidate[]>("candidates")) || []; const newCandidate: Candidate = { id: crypto.randomUUID(), name: name, votes: 0, }; candidates.push(newCandidate); await Zipper.storage.set("candidates", candidates); return newCandidate; } For this use case, we just need to provide a candidate name, but the Candidate object contains a unique ID and the number of votes received. While here, I went ahead and wrote the delete-all-candidates.ts file, which removes all candidates from the key/value data store: TypeScript import { validateRequest } from "./common.ts"; type Input = { force: boolean; }; export async function handler( { force }: Input, context: Zipper.HandlerContext ) { validateRequest(context); if (force) { await Zipper.storage.set("candidates", []); } } At this point, I used the Preview functionality to create Candidate A, Candidate B, and Candidate C: Registering Voters With the ballot ready, I needed the ability to register voters for the ballot. So I added a create-voter.ts file with the following content: TypeScript import { Voter } from "./types.ts"; import { validateRequest } from "./common.ts"; type Input = { email: string; name: string; }; export async function handler( { email, name }: Input, context: Zipper.HandlerContext ) { validateRequest(context); const voters = (await Zipper.storage.get<Voter[]>("voters")) || []; const newVoter: Voter = { email: email, name: name, voted: false, }; voters.push(newVoter); await Zipper.storage.set("voters", voters); return newVoter; } To register a voter, I decided to provide inputs for email address and name. There is also a boolean property called voted which will be used to enforce the vote-only-once rule. Like before, I went ahead and created the delete-all-voters.ts file: TypeScript import { validateRequest } from "./common.ts"; type Input = { force: boolean; }; export async function handler( { force }: Input, context: Zipper.HandlerContext ) { validateRequest(context); if (force) { await Zipper.storage.set("voters", []); } } Now that we were ready to register some voters, I registered myself as a voter for the ballot: Creating the Ballot The last thing I needed to do was establish the ballot. This involved updating the main.ts as shown below: TypeScript import { Constants } from "./constants.ts"; import { Candidate, Voter } from "./types.ts"; type Input = { email: string; }; export async function handler({ email }: Input) { const voters = (await Zipper.storage.get<Voter[]>("voters")) || []; const voter = voters.find((v) => v.email == email); const candidates = (await Zipper.storage.get<Candidate[]>("candidates")) || []; if (email && voter && candidates.length > 0) { return { candidates: candidates.map((candidate) => { return { Candidate: candidate.name, Votes: candidate.votes, actions: [ Zipper.Action.create({ actionType: "button", showAs: "refresh", path: "vote", text: `Vote for ${candidate.name}`, isDisabled: voter.voted, inputs: { candidateId: candidate.id, voterId: voter.email, }, }), ], }; }), }; } else if (!email) { <> <h4>Error:</h4> <p> You must provide a valid email address in order to vote for this ballot. </p> </>; } else if (!voter) { return ( <> <h4>Invalid Email Address:</h4> <p> The email address provided ({email}) is not authorized to vote for this ballot. </p> </> ); } else { return ( <> <h4>Ballot Not Ready:</h4> <p>No candidates have been configured for this ballot.</p> <p>Please try again later.</p> </> ); } } export const config: Zipper.HandlerConfig = { description: { title: Constants.BALLOT_TITLE, subtitle: Constants.BALLOT_SUBTITLE, }, }; I added the following validations as part of the processing logic: The email property must be included or else a “You must provide a valid email address in order to vote for this ballot” message will be displayed. The email value provided must match a registered voter or else a “The email address provided is not authorized to vote for this ballot” message will be displayed. There must be at least one candidate to vote on or else a “No candidates have been configured for this ballot” message will be displayed. If the registered voter has already voted, the voting buttons will be disabled for all candidates on the ballot. The main.ts file contains a button for each candidate, all of which call the vote.ts file, displayed below: TypeScript import { Candidate, Voter } from "./types.ts"; type Input = { candidateId: string; voterId: string; }; export async function handler({ candidateId, voterId }: Input) { const candidates = (await Zipper.storage.get<Candidate[]>("candidates")) || []; const candidate = candidates.find((c) => c.id == candidateId); const candidateIndex = candidates.findIndex(c => c.id == candidateId); const voters = (await Zipper.storage.get<Voter[]>("voters")) || []; const voter = voters.find((v) => v.email == voterId); const voterIndex = voters.findIndex(v => v.email == voterId); if (candidate && voter) { candidate.votes++; candidates[candidateIndex] = candidate; voter.voted = true; voters[voterIndex] = voter; await Zipper.storage.set("candidates", candidates); await Zipper.storage.set("voters", voters); return `${voter.name} successfully voted for ${candidate.name}`; } return `Could not vote. candidate=${ candidate }, voter=${ voter }`; } At this point, the ballot applet was ready for use. HOA Ballot In Action For each registered voter, I would send them an email with a link similar to what is listed below: https://squeeking-echoing-cricket.zipper.run/run/main.ts?email=some.email@example.com The link would be customized to provide the appropriate email address for the email query parameter. Clicking the link runs the main.ts file and passes in the email parameter, avoiding the need for the registered voter to have to type in their email address. The ballot appears as shown below: I decided to cast my vote for Candidate B. Once I pushed the button, the ballot was updated as shown: The number of votes for Candidate B increased by one, and all of the voting buttons were disabled. Success! Conclusion Looking back on the requirements for the ballot applet, I realized I was able to meet all of the criteria, including the stretch goals in about two hours—and this included having a UI, infrastructure, and deployment. The best part of this experience was that 100% of my time was focused on building my solution, and I didn’t need to spend any time dealing with infrastructure or even the persistence store. My readers may recall that I have been focused on the following mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” - J. Vester The Zipper platform adheres to my personal mission statement 100%. In fact, they have been able to take things a step further than Ruby on Rails did, because I don’t have to worry about where my service will run or what data store I will need to configure. Using the applet approach, my ballot is already deployed and ready for use. If you are interested in giving applets a try, simply login to zipper.dev and start building. Currently, using the Zipper platform is free. Give the AI-Generated Code template a try, as it is really cool to provide a paragraph of what you want to build and see how closely the resulting applet matches what you have in mind. If you want to give my ballot applet a try, it is also available to fork in the Zipper gallery. Have a really great day!

Verification and validation are two distinct processes often used in various fields, including software development, engineering, and manufacturing. They are both used to ensure that the software meets its intended purpose, but they do so in different ways. Verification Verification is the process of checking whether the software meets its specifications. It answers the question: "Are we building the product right?" This means checking that the software does what it is supposed to do, according to the requirements that were defined at the start of the project. Verification is typically done by static testing, which means that the software is not actually executed. Instead, the code is reviewed, inspected, or walked through to ensure that it meets the specifications. Validation Validation is the process of checking whether the software meets the needs of its users. It answers the question: "Are we building the right product?" This means checking that the software is actually useful and meets the expectations of the people who will be using it. Validation is typically done by dynamic testing, which means that the software is actually executed and tested with real data. Here are some typical examples of verification and validation: Verification: Checking the code of a software program to make sure that it follows the correct syntax and that all of the functions are implemented correctly Validation: Testing a software program with real data to make sure that it produces the correct results Verification: Reviewing the design documents for a software system to make sure that they are complete and accurate Validation: Conducting user acceptance testing (UAT) to make sure that a software system meets the needs of its users When To Use Conventionally, verification should be done early in the software development process, while validation should be done later. This is because verification can help to identify and fix errors early on, which can save time and money in the long run. Validation is also important, but it can be done after the software is mostly complete since it involves real-world testing and feedback. Another approach would be to start verification and validation as early as possible and iterate. Small, incremental verification steps can be followed by validation whenever possible. Such iterations between verification and validation can be used throughout the development phase. The reasoning behind this approach is that both verification and validation may help to identify and fix errors early. Weather Forecasting App Imagine a team of software engineers developing a weather forecasting app. They have a specification that states, "The app should display the current temperature and a 5-day weather forecast accurately." During the testing phase, they meticulously review the code, check the algorithms, and ensure that the app indeed displays the temperature and forecast data correctly according to their specifications. If everything aligns with the specification, the app passes verification because it meets the specified criteria. Now, let's shift our focus to the users of this weather app. They download the app, start using it, and provide feedback. Some users report that while the temperature and forecasts are accurate, they find the user interface confusing and difficult to navigate. Others suggest that the app should provide more detailed hourly forecasts. This feedback pertains to the user experience and user satisfaction, rather than specific technical specifications. Verification confirms that the app meets the technical requirements related to temperature and forecast accuracy, but validation uncovers issues with the user interface and user needs. The app may pass verification but fail validation because it doesn't fully satisfy the true needs and expectations of its users. This highlights that validation focuses on whether the product meets the actual needs and expectations of the users, which may not always align with the initial technical specifications. Social Media App Let's say you are developing a new social media app. The verification process would involve ensuring that the app meets the specified requirements, such as the ability to create and share posts, send messages, and add friends. This could be done by reviewing the app's code, testing its features, and comparing it to the requirements document. The validation process would involve ensuring that the app meets the needs of the users. This could be done by conducting user interviews, surveys, and usability testing. For example, you might ask users how they would like to be able to share posts, or what features they would like to see added to the app. In this example, verification would ensure that the app is technically sound, while validation would ensure that it is user-friendly and meets the needs of the users. Online Payment Processing App A team of software engineers is developing an online payment processing app. For verification, they would verify that the code for processing payments, calculating transaction fees, and handling currency conversions has been correctly implemented according to the app's design specifications. They would also ensure that the app adheres to industry security standards, such as the Payment Card Industry Data Security Standard (PCI DSS), by verifying that encryption protocols, access controls, and authentication mechanisms are correctly integrated. They would also confirm that the user interface functions as intended, including verifying that the payment forms collect necessary information and that error messages are displayed appropriately. To validate the online payment processing software, they would use it in actual payment transactions. One case would be to process real payment transactions to confirm that the software can handle various types of payments, including credit cards, digital wallets, and international transactions, without errors. Another case would be to evaluate the user experience, checking if users can easily navigate the app, make payments, and receive confirmation without issues. Predicting Brain Activity Using fMRI A neuroinformatics software app is developed to predict brain activity based on functional magnetic resonance imaging (fMRI) data. Verification would verify that the algorithms used for preprocessing fMRI data, such as noise removal and motion correction, are correctly translated into code. You would also ensure that the user interface functions as specified, and that data input and output formats adhere to the defined standards, such as the Brain Imaging Data Structure (BIDS). Validation would compare the predicted brain activity patterns generated by the software to the actual brain activity observed in the fMRI scans. Additionally, you might compare the software's predictions to results obtained using established methods or ground truth data to evaluate its accuracy. Validation in this context ensures that the software not only runs without internal errors (as verified) but also that it reliably and accurately performs its primary function of predicting brain activity based on fMRI data. This step helps determine if the software can be trusted for scientific or clinical purposes. Predicting the Secondary Structure of RNA Molecules Imagine you are a bioinformatician working on a software tool that predicts the secondary structure of RNA molecules. Your software takes an RNA sequence as input and predicts the most likely folding pattern. For verification, you want to verify that your RNA secondary structure prediction software calculates free energy values accurately using the algorithms described in the scientific literature. You compare the software's implementation against the published algorithms and validate that the code follows the expected mathematical procedures precisely. In this context, verification ensures that your software performs the intended computations correctly and follows the algorithmic logic accurately. To validate your RNA secondary structure prediction software, you would run it on a diverse set of real-world RNA sequences with known secondary structures. You would then compare the software's predictions against experimental data or other trusted reference tools to check if it provides biologically meaningful results and if its accuracy is sufficient for its intended purpose. The Light Switch in a Conference Room Consider a light switch in a conference room. Verification asks whether the lighting meets the requirements. The requirements might state that "the lights in front of the projector screen can be controlled independently of the other lights in the room." If the requirements are written down and the lights cannot be controlled independently, then the lighting fails verification. This is because the implementation does not meet the requirements. Validation asks whether the users are satisfied with the lighting. This is a more subjective question, and it is not always easy to measure satisfaction with a single metric. For example, even if the lights can be controlled independently, the users may still be dissatisfied if the lights are too bright or too dim. Wrapping Up Verification is usually a more technical activity that uses knowledge about software artifacts, requirements, and specifications. Validation usually depends on domain knowledge, that is, knowledge of the application for which the software is written. For example, validation of medical device software requires knowledge from healthcare professionals, clinicians, and patients. It is important to note that verification and validation are not mutually exclusive. In fact, they are complementary processes. Verification ensures that the software is built correctly, while validation ensures that the software is useful. By combining verification and validation, we can be more confident that our product will make customers happy.

This is an article from DZone's 2023 Automated Testing Trend Report.For more: Read the Report Artificial intelligence (AI) has revolutionized the realm of software testing, introducing new possibilities and efficiencies. The demand for faster, more reliable, and efficient testing processes has grown exponentially with the increasing complexity of modern applications. To address these challenges, AI has emerged as a game-changing force, revolutionizing the field of automated software testing. By leveraging AI algorithms, machine learning (ML), and advanced analytics, software testing has undergone a remarkable transformation, enabling organizations to achieve unprecedented levels of speed, accuracy, and coverage in their testing endeavors. This article delves into the profound impact of AI on automated software testing, exploring its capabilities, benefits, and the potential it holds for the future of software quality assurance. An Overview of AI in Testing This introduction aims to shed light on the role of AI in software testing, focusing on key aspects that drive its transformative impact. Figure 1: AI in testing Elastically Scale Functional, Load, and Performance Tests AI-powered testing solutions enable the effortless allocation of testing resources, ensuring optimal utilization and adaptability to varying workloads. This scalability ensures comprehensive testing coverage while maintaining efficiency. AI-Powered Predictive Bots AI-powered predictive bots are a significant advancement in software testing. Bots leverage ML algorithms to analyze historical data, patterns, and trends, enabling them to make informed predictions about potential defects or high-risk areas. By proactively identifying potential issues, predictive bots contribute to more effective and efficient testing processes. Automatic Update of Test Cases With AI algorithms monitoring the application and its changes, test cases can be dynamically updated to reflect modifications in the software. This adaptability reduces the effort required for test maintenance and ensures that the test suite remains relevant and effective over time. AI-Powered Analytics of Test Automation Data By analyzing vast amounts of testing data, AI-powered analytical tools can identify patterns, trends, and anomalies, providing valuable information to enhance testing strategies and optimize testing efforts. This data-driven approach empowers testing teams to make informed decisions and uncover hidden patterns that traditional methods might overlook. Visual Locators Visual locators, a type of AI application in software testing, focus on visual elements such as user interfaces and graphical components. AI algorithms can analyze screenshots and images, enabling accurate identification of and interaction with visual elements during automated testing. This capability enhances the reliability and accuracy of visual testing, ensuring a seamless user experience. Self-Healing Tests AI algorithms continuously monitor test execution, analyzing results and detecting failures or inconsistencies. When issues arise, self-healing mechanisms automatically attempt to resolve the problem, adjusting the test environment or configuration. This intelligent resilience minimizes disruptions and optimizes the overall testing process. What Is AI-Augmented Software Testing? AI-augmented software testing refers to the utilization of AI techniques — such as ML, natural language processing, and data analytics — to enhance and optimize the entire software testing lifecycle. It involves automating test case generation, intelligent test prioritization, anomaly detection, predictive analysis, and adaptive testing, among other tasks. By harnessing the power of AI, organizations can improve test coverage, detect defects more efficiently, reduce manual effort, and ultimately deliver high-quality software with greater speed and accuracy. Benefits of AI-Powered Automated Testing AI-powered software testing offers a plethora of benefits that revolutionize the testing landscape. One significant advantage lies in its codeless nature, thus eliminating the need to memorize intricate syntax. Embracing simplicity, it empowers users to effortlessly create testing processes through intuitive drag-and-drop interfaces. Scalability becomes a reality as the workload can be efficiently distributed among multiple workstations, ensuring efficient utilization of resources. The cost-saving aspect is remarkable as minimal human intervention is required, resulting in substantial reductions in workforce expenses. With tasks executed by intelligent bots, accuracy reaches unprecedented heights, minimizing the risk of human errors. Furthermore, this automated approach amplifies productivity, enabling testers to achieve exceptional output levels. Irrespective of the software type — be it a web-based desktop application or mobile application — the flexibility of AI-powered testing seamlessly adapts to diverse environments, revolutionizing the testing realm altogether. Figure 2: Benefits of AI for test automation Mitigating the Challenges of AI-Powered Automated Testing AI-powered automated testing has revolutionized the software testing landscape, but it is not without its challenges. One of the primary hurdles is the need for high-quality training data. AI algorithms rely heavily on diverse and representative data to perform effectively. Therefore, organizations must invest time and effort in curating comprehensive and relevant datasets that encompass various scenarios, edge cases, and potential failures. Another challenge lies in the interpretability of AI models. Understanding why and how AI algorithms make specific decisions can be critical for gaining trust and ensuring accurate results. Addressing this challenge requires implementing techniques such as explainable AI, model auditing, and transparency. Furthermore, the dynamic nature of software environments poses a challenge in maintaining AI models' relevance and accuracy. Continuous monitoring, retraining, and adaptation of AI models become crucial to keeping pace with evolving software systems. Additionally, ethical considerations, data privacy, and bias mitigation should be diligently addressed to maintain fairness and accountability in AI-powered automated testing. AI models used in testing can sometimes produce false positives (incorrectly flagging a non-defect as a defect) or false negatives (failing to identify an actual defect). Balancing precision and recall of AI models is important to minimize false results. AI models can exhibit biases and may struggle to generalize new or uncommon scenarios. Adequate training and validation of AI models are necessary to mitigate biases and ensure their effectiveness across diverse testing scenarios. Human intervention plays a critical role in designing test suites by leveraging their domain knowledge and insights. They can identify critical test cases, edge cases, and scenarios that require human intuition or creativity, while leveraging AI to handle repetitive or computationally intensive tasks. Continuous improvement would be possible by encouraging a feedback loop between human testers and AI systems. Human experts can provide feedback on the accuracy and relevance of AI-generated test cases or predictions, helping improve the performance and adaptability of AI models. Human testers should play a role in the verification and validation of AI models, ensuring that they align with the intended objectives and requirements. They can evaluate the effectiveness, robustness, and limitations of AI models in specific testing contexts. AI-Driven Testing Approaches AI-driven testing approaches have ushered in a new era in software quality assurance, revolutionizing traditional testing methodologies. By harnessing the power of artificial intelligence, these innovative approaches optimize and enhance various aspects of testing, including test coverage, efficiency, accuracy, and adaptability. This section explores the key AI-driven testing approaches, including differential testing, visual testing, declarative testing, and self-healing automation. These techniques leverage AI algorithms and advanced analytics to elevate the effectiveness and efficiency of software testing, ensuring higher-quality applications that meet the demands of the rapidly evolving digital landscape: Differential testing assesses discrepancies between application versions and builds, categorizes the variances, and utilizes feedback to enhance the classification process through continuous learning. Visual testing utilizes image-based learning and screen comparisons to assess the visual aspects and user experience of an application, thereby ensuring the integrity of its look and feel. Declarative testing expresses the intention of a test using a natural or domain-specific language, allowing the system to autonomously determine the most appropriate approach to execute the test. Self-healing automation automatically rectifies element selection in tests when there are modifications to the user interface (UI), ensuring the continuity of reliable test execution. Key Considerations for Harnessing AI for Software Testing Many contemporary test automation tools infused with AI provide support for open-source test automation frameworks such as Selenium and Appium. AI-powered automated software testing encompasses essential features such as auto-code generation and the integration of exploratory testing techniques. Open-Source AI Tools To Test Software When selecting an open-source testing tool, it is essential to consider several factors. Firstly, it is crucial to verify that the tool is actively maintained and supported. Additionally, it is critical to assess whether the tool aligns with the skill set of the team. Furthermore, it is important to evaluate the features, benefits, and challenges presented by the tool to ensure they are in line with your specific testing requirements and organizational objectives. A few popular open-source options include, but are not limited to: Carina – AI-driven, free forever, scriptless approach to automate functional, performance, visual, and compatibility tests TestProject – Offered the industry's first free Appium AI tools in 2021, expanding upon the AI tools for Selenium that they had previously introduced in 2020 for self-healing technology Cerberus Testing – A low-code and scalable test automation solution that offers a self-healing feature called Erratum and has a forever-free plan Designing Automated Tests With AI and Self-Testing AI has made significant strides in transforming the landscape of automated testing, offering a range of techniques and applications that revolutionize software quality assurance. Some of the prominent techniques and algorithms are provided in the tables below, along with the purposes they serve: KEY TECHNIQUES AND APPLICATIONS OF AI IN AUTOMATED TESTING Key Technique Applications Machine learning Analyze large volumes of testing data, identify patterns, and make predictions for test optimization, anomaly detection, and test case generation Natural language processing Facilitate the creation of intelligent chatbots, voice-based testing interfaces, and natural language test case generation Computer vision Analyze image and visual data in areas such as visual testing, UI testing, and defect detection Reinforcement learning Optimize test execution strategies, generate adaptive test scripts, and dynamically adjust test scenarios based on feedback from the system under test Table 1 KEY ALGORITHMS USED FOR AI-POWERED AUTOMATED TESTING Algorithm Purpose Applications Clustering algorithms Segmentation k-means and hierarchical clustering are used to group similar test cases, identify patterns, and detect anomalies Sequence generation models: recurrent neural networks or transformers Text classification and sequence prediction Trained to generate sequences such as test scripts or sequences of user interactions for log analysis Bayesian networks Dependencies and relationships between variables Test coverage analysis, defect prediction, and risk assessment Convolutional neural networks Image analysis Visual testing Evolutionary algorithms: genetic algorithms Natural selection Optimize test case generation, test suite prioritization, and test execution strategies by applying genetic operators like mutation and crossover on existing test cases to create new variants, which are then evaluated based on fitness criteria Decision trees, fandom forests, support vector machines, and neural networks Classification Classification of software components Variational autoencoders and generative adversarial networks Generative AI Used to generate new test cases that cover different scenarios or edge cases by test data generation, creating synthetic data that resembles real-world scenarios Table 2 Real-World Examples of AI-Powered Automated Testing AI-powered visual testing platforms perform automated visual validation of web and mobile applications. They use computer vision algorithms to compare screenshots and identify visual discrepancies, enabling efficient visual testing across multiple platforms and devices. NLP and ML are combined to generate test cases from plain English descriptions. They automatically execute these test cases, detect bugs, and provide actionable insights to improve software quality. Self-healing capabilities are also provided by automatically adapting test cases to changes in the application's UI, improving test maintenance efficiency. Quantum AI-Powered Automated Testing: The Road Ahead The future of quantum AI-powered automated software testing holds great potential for transforming the way testing is conducted. Figure 3: Transition of automated testing from AI to Quantum AI Quantum computing's ability to handle complex optimization problems can significantly improve test case generation, test suite optimization, and resource allocation in automated testing. Quantum ML algorithms can enable more sophisticated and accurate models for anomaly detection, regression testing, and predictive analytics. Quantum computing's ability to perform parallel computations can greatly accelerate the execution of complex test scenarios and large-scale test suites. Quantum algorithms can help enhance security testing by efficiently simulating and analyzing cryptographic algorithms and protocols. Quantum simulation capabilities can be leveraged to model and simulate complex systems, enabling more realistic and comprehensive testing of software applications in various domains, such as finance, healthcare, and transportation. Parting Thoughts AI has significantly revolutionized the traditional landscape of testing, enhancing the effectiveness, efficiency, and reliability of software quality assurance processes. AI-driven techniques such as ML, anomaly detection, NLP, and intelligent test prioritization have enabled organizations to achieve higher test coverage, early defect detection, streamlined test script creation, and adaptive test maintenance. The integration of AI in automated testing not only accelerates the testing process but also improves overall software quality, leading to enhanced customer satisfaction and reduced time to market. As AI continues to evolve and mature, it holds immense potential for further advancements in automated testing, paving the way for a future where AI-driven approaches become the norm in ensuring the delivery of robust, high-quality software applications. Embracing the power of AI in automated testing is not only a strategic imperative but also a competitive advantage for organizations looking to thrive in today's rapidly evolving technological landscape. This is an article from DZone's 2023 Automated Testing Trend Report.For more: Read the Report

This is an article from DZone's 2023 Automated Testing Trend Report.For more: Read the Report One of the core capabilities that has seen increased interest in the DevOps community is observability. Observability improves monitoring in several vital ways, making it easier and faster to understand business flows and allowing for enhanced issue resolution. Furthermore, observability goes beyond an operations capability and can be used for testing and quality assurance. Testing has traditionally faced the challenge of identifying the appropriate testing scope. "How much testing is enough?" and "What should we test?" are questions each testing executive asks, and the answers have been elusive. There are fewer arguments about testing new functionality; while not trivial, you know the functionality you built in new features and hence can derive the proper testing scope from your understanding of the functional scope. But what else should you test? What is a comprehensive general regression testing suite, and what previous functionality will be impacted by the new functionality you have developed and will release? Observability can help us with this as well as the unavoidable defect investigation. But before we get to this, let's take a closer look at observability. What Is Observability? Observability is not monitoring with a different name. Monitoring is usually limited to observing a specific aspect of a resource, like disk space or memory of a compute instance. Monitoring one specific characteristic can be helpful in an operations context, but it usually only detects a subset of what is concerning. All monitoring can show is that the system looks okay, but users can still be experiencing significant outages. Observability aims to make us see the state of the system by making data flows "observable." This means that we can identify when something starts to behave out of order and requires our attention. Observability combines logs, metrics, and traces from infrastructure and applications to gain insights. Ideally, it organizes these around workflows instead of system resources and, as such, creates a functional view of the system in use. Done correctly, it lets you see what functionality is being executed and how frequently, and it enables you to identify performance characteristics of the system and workflow. Figure 1: Observability combines metrics, logs, and traces for insights One benefit of observability is that it shows you the actual system. It is not biased by what the designers, architects, and engineers think should happen in production. It shows the unbiased flow of data. The users, over time (and sometimes from the very first day), find ways to use the system quite differently from what was designed. Observability makes such changes in behavior visible. Observability is incredibly powerful in debugging system issues as it allows us to navigate the system to see where problems occur. Observability requires a dedicated setup and some contextual knowledge similar to traceability. Traceability is the ability to follow a system transaction over time through all the different components of our application and infrastructure architecture, which means you have to have common information like an ID that enables this. OpenTelemetry is an open standard that can be used and provides useful guidance on how to set this up. Observability makes identifying production issues a lot easier. And we can use observability for our benefit in testing, too. Observability of Testing: How to Look Left Two aspects of observability make it useful in the testing context: Its ability to make the actual system usage observable and its usefulness in finding problem areas during debugging. Understanding the actual system behavior is most directly useful during performance testing. Performance testing is the pinnacle of testing since it tries to achieve as close to the realistic peak behavior of a system as possible. Unfortunately, performance testing scenarios are often based on human knowledge of the system instead of objective information. For example, performance testing might be based on the prediction of 10,000 customer interactions per hour during a sales campaign based on the information of the sales manager. Observability information can help define the testing scenarios by using the information to look for the times the system was under the most stress in production and then simulate similar situations in the performance test environment. We can use a system signature to compare behaviors. A system signature in the context of observability is the set of values for logs, metrics, and traces during a specific period. Take, for example, a marketing promotion for new customers. The signature of the system should change during that period to show more new account creations with its associated functionality and the related infrastructure showing up as being more "busy." If the signature does not change during the promotion, we would predict that we also don't see the business metrics move (e.g., user sign-ups). In this example, the business metrics and the signature can be easily matched. Figure 2: A system behaving differently in test, which shows up in the system signature In many other cases, this is not true. Imagine an example where we change the recommendation engine to use our warehouse data going forward. We expect the system signature to show increased data flows between the recommendation engine and our warehouse system. You can see how system signatures and the changes of the system signature can be useful for testing; any differences in signature between production and the testing systems should be explainable by the intended changes of the upcoming release. Otherwise, investigation is required. In the same way, information from the production observability system can be used to define a regression suite that reflects the functionality most frequently used in production. Observability can give you information about the workflows still actively in use and which workflows have stopped being relevant. This information can optimize your regression suite both from a maintenance perspective and, more importantly, from a risk perspective, making sure that core functionality, as experienced by the user, remains in a working state. Implementing observability in your test environments means you can use the power of observability for both production issues and your testing defects. It removes the need for debugging modes to some degree and relies upon the same system capability as production. This way, observability becomes how you work across both dev and ops, which helps break down silos. Observability for Test Insights: Looking Right In the previous section, we looked at using observability by looking left or backward, ensuring we have kept everything intact. Similarly, we can use observability to help us predict the success of the features we deliver. Think about a new feature you are developing. During the test cycles, we see how this new feature changes the workflows, which shows up in our observability solution. We can see the new features being used and other features changing in usage as a result. The signature of our application has changed when we consider the logs, traces, and metrics of our system in test. Once we go live, we predict that the signature of the production system will change in a very similar way. If that happens, we will be happy. But what if the signature of the production system does not change as predicted? Let's take an example: We created a new feature that leverages information from previous bookings to better serve our customers by allocating similar seats and menu options. During testing, we tested the new feature with our test data set, and we see an increase in accessing the bookings database while the customer booking is being collated. Once we go live, we realize that the workflows are not utilizing the customer booking database, and we leverage the information from our observability tooling to investigate. We have found a case where the users are not using our new features or are not using the features in the expected way. In either case, this information allows us to investigate further to see whether more change management is required for the users or whether our feature is just not solving the problem in the way we wanted it to. Another way to use observability is to evaluate the performance of your changes in test and the impact on the system signature — comparing this afterwards with the production system signature can give valuable insights and prevent overall performance degradation. Our testing efforts (and the associated predictions) have now become a valuable tool for the business to evaluate the success of a feature, which elevates testing to become a business tool and a real value investment. Figure 3: Using observability in test by looking left and looking right Conclusion While the popularity of observability is a somewhat recent development, it is exciting to see what benefits it can bring to testing. It will create objectiveness for defining testing efforts and results by evaluating them against the actual system behavior in production. It also provides value to developer, tester, and business communities, which makes it a valuable tool for breaking down barriers. Using the same practices and tools across communities drives a common culture — after all, culture is nothing but repeated behaviors. This is an article from DZone's 2023 Automated Testing Trend Report.For more: Read the Report