Software Design and Architecture Resources

DZone's Featured Software Design and Architecture Resources

Scaling in Practice: Caching and Rate-Limiting With Redis and Next.js

By Ernest Nnamdi

As someone who loves shipping products and tools for people to experiment with and use, it was only a matter of time before scalability issues caught up with me. I always knew what scaling meant and how to scale in theory. I was familiar with vertical and horizontal scaling, and the intricacies of each, but again, in theory. I had actually never needed to implement any of them, as my user base was usually modest. This is until I built a faucet that exceeded my expectations in terms of usage. But this massive traction came with its own set of issues. The TL;DR of the issue was that even though I restricted the frequency with which a wallet address could claim funds to once a day, a user could simply spin up new addresses and keep claiming. This resulted in my faucet being drained frequently. To solve this issue, I had to introduce rate limiting to restrict the IP address of each user from abusing the faucet. Similarly, in the LMS platform I built, I had to introduce caching to reduce the number of calls to the database. In this tutorial, I am going to demonstrate to you how you can implement rate limiting and caching in your application using Redis. Why Redis? Redis is like that reliable friend who's surprisingly good at everything. Need lightning-fast data retrieval? Redis. Need to coordinate between multiple servers? Redis. Need to limit how often users can perform actions? You guessed it — Redis. We can cache API responses, track rate limits, and generally make our applications much more resilient. The best part? It's actually quite simple to implement once you understand the patterns. That being said, let's get to building! Prerequisites Set up a Redis Cloud account and get the public URLFamiliarity with Next.js What Are We Building? We'll create a Next.js application with: A product catalog API with Redis-powered cachingA flexible rate-limiting system using RedisInteractive demo pages to visualize both techniques in action Here is a preview of what our project structure will look like: Plain Text /src /lib - redis.ts # Redis client and caching utilities - edge-rate-limiter.ts # Edge-compatible rate limiting - product.ts # Product data service /app /api /products # Product data API endpoints /cache/invalidate # Cache management endpoint /tests # Test endpoints for rate limiting /demo # Demo UI pages /caching # Redis caching demo /rate-limiting # Rate limiting demo - middleware.ts # Rate limiting middleware Step 1: Setting Up Our Environment Let's start by creating a new Next.js project. Plain Text npx create-next-app redis-scaling-demo cd redis-scaling-demo For the accompanying prompts, select the default options and hit Enter. Next, we install our Redis dependencies and also the types. Plain Text npm install ioredis @upstash/ratelimit @upstash/redis @types/ioredis --save-dev If you do not already have a Redis Cloud account, go to Redis, create a new account, and select the free database. Look for the button above and click it to select the CLI option. Copy the command that contains your host and port. Plain Text redis://default:password@host:port It should look like the above format (note: ignore the "redis-cli -u" in the command). Finally, replace the password(the asterisks in the command) with your actual password, which you can get by going to the security section and copying the password. Finally, create a .env file in the root of your project(assuming you have opened it up in your code terminal) and paste in the redis url like so: TypeScript REDIS_URL=redis://default:password@host:port Step 2: Creating Services In the src folder, create a new folder called lib and then a file called redis.ts. Paste in the code below. TypeScript import Redis from "ioredis"; let redisClient: Redis | null = null; export function getRedisClient() { if (!redisClient) { redisClient = new Redis(process.env.REDIS_URL || "redis://localhost:6379", { connectTimeout: 10000, // 10 seconds retryStrategy(times) { const delay = Math.min(times * 50, 2000); return delay; }, maxRetriesPerRequest: 3, }); redisClient.on("error", (err) => { console.error("Redis connection error:", err); }); redisClient.on("connect", () => { console.log("Successfully connected to Redis"); }); } return redisClient; } function createCacheKey(prefix: string, key: string): string { return `${prefix}${key}`; } async function getCachedData<T>( redis: Redis, prefix: string, key: string ): Promise<T | null> { const cachedData = await redis.get(createCacheKey(prefix, key)); if (!cachedData) { return null; } try { return JSON.parse(cachedData) as T; } catch (error) { console.error(`Error parsing cached data for key ${key}:`, error); return null; } } async function setCachedData<T>( redis: Redis, prefix: string, key: string, data: T, ttl: number ): Promise<void> { await redis.set(createCacheKey(prefix, key), JSON.stringify(data), "EX", ttl); } async function deleteCachedData( redis: Redis, prefix: string, key: string ): Promise<void> { await redis.del(createCacheKey(prefix, key)); } async function invalidateCachePattern( redis: Redis, prefix: string, pattern: string ): Promise<number> { const keys = await redis.keys(`${prefix}${pattern}*`); if (keys.length === 0) { return 0; } return await redis.del(...keys); } export function getCacheInstance( prefix: string = "cache:", defaultTTL: number = 60 * 5 ) { const redis = getRedisClient(); return { get: <T>(key: string) => getCachedData<T>(redis, prefix, key), set: <T>(key: string, data: T, ttl?: number) => setCachedData<T>(redis, prefix, key, data, ttl || defaultTTL), delete: (key: string) => deleteCachedData(redis, prefix, key), invalidatePattern: (pattern: string) => invalidateCachePattern(redis, prefix, pattern), }; } Next.js middleware runs in the Edge Runtime, which has limitations compared to Node.js. We need to create a rate limiter that works in this environment, so in the lib folder, create another file called edge-rate-limiter.ts. TypeScript import { NextRequest, NextResponse } from "next/server"; // Simple in-memory store (note: this won't work in a multi-instance setup) const ipRequests = new Map<string, { count: number; timestamp: number }>(); export function createEdgeRateLimiter(options: { limit: number; timeWindow: number; }) { return function rateLimit(request: NextRequest) { const ip = request.headers.get("x-forwarded-for")?.split(",")[0] || request.headers.get("x-real-ip") || "127.0.0.1"; const now = Date.now(); const windowStart = now - options.timeWindow * 1000; const currentData = ipRequests.get(ip) || { count: 0, timestamp: now }; if (currentData.timestamp < windowStart) { currentData.count = 0; currentData.timestamp = now; } currentData.count++; ipRequests.set(ip, currentData); const response = NextResponse.next(); response.headers.set("X-RateLimit-Limit", options.limit.toString()); response.headers.set( "X-RateLimit-Remaining", Math.max(0, options.limit - currentData.count).toString() ); if (currentData.count > options.limit) { return NextResponse.json({ error: "Too many requests" }, { status: 429 }); } return response; }; } For rate limiting, we create a rate-limiter.ts file in the lib folder. TypeScript import { NextRequest, NextResponse } from "next/server"; import { getRedisClient } from "./redis"; import Redis from "ioredis"; export interface RateLimitConfig { maxRequests: number; windowSizeInSeconds: number; prefix: string; } function getRedisKey(prefix: string, identifier: string): string { return `${prefix}:${identifier}`; } export async function checkRateLimit( redis: Redis, config: RateLimitConfig, identifier: string ): Promise<{ success: boolean; limit: number; remaining: number; reset: number; }> { const key = getRedisKey(config.prefix, identifier); const now = Math.floor(Date.now() / 1000); const windowExpiry = now + config.windowSizeInSeconds; const multi = redis.multi(); multi.incr(key); multi.expire(key, config.windowSizeInSeconds); multi.get(key); multi.ttl(key); const results = await multi.exec(); if (!results) { throw new Error("Redis transaction failed"); } const requestCount = parseInt(results[2][1] as string) || 1; const ttl = parseInt(results[3][1] as string) || config.windowSizeInSeconds; const resetTime = now + ttl; const remainingRequests = Math.max(0, config.maxRequests - requestCount); return { success: requestCount <= config.maxRequests, limit: config.maxRequests, remaining: remainingRequests, reset: resetTime, }; } export function getClientIdentifier(req: NextRequest): string { const forwardedFor = req.headers.get("x-forwarded-for") || ""; const realIp = req.headers.get("x-real-ip"); const cfConnectingIp = req.headers.get("cf-connecting-ip"); if (forwardedFor && forwardedFor.includes(",")) { return forwardedFor.split(",")[0].trim(); } return cfConnectingIp || realIp || forwardedFor || "127.0.0.1"; } export function createRateLimitMiddleware(config: RateLimitConfig) { return async function rateLimitMiddleware( req: NextRequest ): Promise<NextResponse> { try { const redis = getRedisClient(); const identifier = getClientIdentifier(req); const { success, limit, remaining, reset } = await checkRateLimit( redis, config, identifier ); const headers = { "X-RateLimit-Limit": limit.toString(), "X-RateLimit-Remaining": remaining.toString(), "X-RateLimit-Reset": reset.toString(), }; if (!success) { return NextResponse.json( { error: "Too Many Requests", message: `Rate limit of ${limit} requests per ${config.windowSizeInSeconds} seconds exceeded.`, }, { status: 429, headers: { ...headers, "Retry-After": Math.ceil( reset - Math.floor(Date.now() / 1000) ).toString(), }, } ); } const response = NextResponse.next(); Object.entries(headers).forEach(([key, value]) => { response.headers.set(key, value); }); return response; } catch (error) { console.error("Rate limit error:", error); return NextResponse.next(); } }; } export const rateLimiters = { default: createRateLimitMiddleware({ maxRequests: 60, windowSizeInSeconds: 60, prefix: "ratelimit:default", }), auth: createRateLimitMiddleware({ maxRequests: 10, windowSizeInSeconds: 60, prefix: "ratelimit:auth", }), sensitive: createRateLimitMiddleware({ maxRequests: 3, windowSizeInSeconds: 60, prefix: "ratelimit:sensitive", }), }; Finally, we create a product service to simulate a database. So, create a product.ts file in the lib folder. TypeScript export interface Product { id: string; name: string; description: string; price: number; category: string; image: string; stock: number; } const productData: Product[] = [ { id: "prod_001", name: "Ergonomic Office Chair", description: "A comfortable chair designed for long working hours", price: 299.99, category: "furniture", image: "/images/chair.jpg", stock: 25, }, { id: "prod_002", name: "Mechanical Keyboard", description: "Mechanical keyboard with RGB lighting and Cherry MX switches", price: 129.99, category: "electronics", image: "/images/keyboard.jpg", stock: 40, }, { id: "prod_003", name: "Wireless Mouse", description: "High-precision wireless mouse with long battery life", price: 49.99, category: "electronics", image: "/images/mouse.jpg", stock: 75, }, { id: "prod_004", name: "Adjustable Standing Desk", description: "Electric adjustable standing desk for better ergonomics", price: 499.99, category: "furniture", image: "/images/desk.jpg", stock: 15, }, { id: "prod_005", name: "Monitor Stand", description: "Adjustable monitor stand with cable management", price: 79.99, category: "accessories", image: "/images/monitor-stand.jpg", stock: 30, }, { id: "prod_006", name: "USB-C Hub", description: "7-in-1 USB-C hub with HDMI, SD card reader, and USB ports", price: 39.99, category: "accessories", image: "/images/usb-hub.jpg", stock: 50, }, { id: "prod_007", name: "Noise-Cancelling Headphones", description: "Over-ear headphones with active noise cancellation", price: 199.99, category: "electronics", image: "/images/headphones.jpg", stock: 20, }, { id: "prod_008", name: "Ergonomic Mouse Pad", description: "Mouse pad with wrist support for ergonomic comfort", price: 19.99, category: "accessories", image: "/images/mousepad.jpg", stock: 100, }, ]; export async function getAllProducts(): Promise<Product[]> { await new Promise((resolve) => setTimeout(resolve, 300)); return [...productData]; } export async function getProductById(id: string): Promise<Product | null> { await new Promise((resolve) => setTimeout(resolve, 200)); return productData.find((product) => product.id === id) || null; } export async function getProductsByCategory( category: string ): Promise<Product[]> { await new Promise((resolve) => setTimeout(resolve, 300)); return productData.filter((product) => product.category === category); } export async function searchProducts(query: string): Promise<Product[]> { await new Promise((resolve) => setTimeout(resolve, 400)); const searchTerm = query.toLowerCase(); return productData.filter( (product) => product.name.toLowerCase().includes(searchTerm) || product.description.toLowerCase().includes(searchTerm) ); } export const productService = { getAllProducts, getProductById, getProductsByCategory, searchProducts, }; Step 3: Rate Limiting — Middleware and Test API We are going to create a middleware for rate limiting, which will basically do three things for us: Creates three different rate limiters with various limitsApplies them based on URL path patternsUses a matcher to only run on API routes Create a middleware.ts file in the root of the src folder and paste in the code below. TypeScript import { NextRequest, NextResponse } from "next/server"; import { createEdgeRateLimiter } from "./lib/edge-rate-limiter"; // Create rate limiters with different settings const rateLimiters = { default: createEdgeRateLimiter({ limit: 60, timeWindow: 60 }), // 60 requests per minute auth: createEdgeRateLimiter({ limit: 10, timeWindow: 60 }), // 10 requests per minute sensitive: createEdgeRateLimiter({ limit: 3, timeWindow: 60 }), // 3 requests per minute }; export function middleware(request: NextRequest) { const path = request.nextUrl.pathname; // Apply different rate limits based on path if (path.startsWith("/api/tests/admin")) { return rateLimiters.sensitive(request); } else if (path.startsWith("/api/tests/auth")) { return rateLimiters.auth(request); } else if (path.startsWith("/api/")) { return rateLimiters.default(request); } return NextResponse.next(); } export const config = { matcher: ["/api/:path*"], }; For the test endpoints to demonstrate different rate limits, create a folder called api in the app directory. Then create another folder called tests. This is where our test endpoints will live. In the tests folder, create three sub-folders: admin, auth and basic respectively. For each of these subfolders, create a route.ts file. In the app/api/tests/admin/route.ts file, paste the code below: TypeScript import { NextRequest, NextResponse } from "next/server"; export function GET(request: NextRequest) { return NextResponse.json({ message: "Admin rate limited endpoint (very strict)", timestamp: new Date().toISOString(), path: "/api/tests/admin", }); } Similarly, in the auth directory: TypeScript import { NextRequest, NextResponse } from "next/server"; export function GET(request: NextRequest) { return NextResponse.json({ message: "Auth rate limited endpoint (moderate)", timestamp: new Date().toISOString(), path: "/api/tests/auth", }); } Finally, in the basic directory: TypeScript import { NextRequest, NextResponse } from "next/server"; export function GET(request: NextRequest) { return NextResponse.json({ message: "Default rate limited endpoint", timestamp: new Date().toISOString(), path: "/api/tests/basic", }); } Step 4: Caching — Building the Product APIs Now we can proceed with creating our api routes for caching. In the app/api folder, create a new folder called products. In this folder, create a sub-folder called categories and a file called route.ts. In the route file, paste in the code below TypeScript import { NextRequest, NextResponse } from "next/server"; import { productService } from "@/lib/product"; import { getCacheInstance } from "@/lib/redis"; // GET /api/products - List products with optional filtering export async function GET(request: NextRequest) { try { // Parse query parameters const searchParams = request.nextUrl.searchParams; const category = searchParams.get("category"); const query = searchParams.get("q"); let cacheKey = "products"; if (category) cacheKey += `:category:${category}`; if (query) cacheKey += `:search:${query}`; const cache = getCacheInstance("api:"); const cachedData = await cache.get(cacheKey); if (cachedData) { console.log(`Cache hit for ${cacheKey}`); const response = NextResponse.json(cachedData); response.headers.set("X-Cache-Status", "HIT"); return response; } console.log(`Cache miss for ${cacheKey}, fetching from database...`); let products; if (query) { products = await productService.searchProducts(query); } else if (category) { products = await productService.getProductsByCategory(category); } else { products = await productService.getAllProducts(); } const responseData = { products, count: products.length, timestamp: new Date().toISOString(), }; await cache.set(cacheKey, responseData, 60 * 5); return NextResponse.json(responseData); } catch (error) { console.error("Error fetching products:", error); return NextResponse.json( { error: "Failed to fetch products" }, { status: 500 } ); } } In the subfolder called categories, create another route.ts file and paste in the below: TypeScript import { NextRequest, NextResponse } from "next/server"; import { productService } from "@/lib/product"; import { getCacheInstance } from "@/lib/redis"; import { Product } from "@/lib/product"; export async function GET(request: NextRequest) { try { const cacheKey = "product:categories"; const cache = getCacheInstance("api:"); const cachedCategories = await cache.get(cacheKey); if (cachedCategories) { console.log(`Cache hit for ${cacheKey}`); return NextResponse.json(cachedCategories); } console.log(`Cache miss for ${cacheKey}, computing categories...`); const products = await productService.getAllProducts(); const categories = Array.from( new Set(products.map((product: Product) => product.category)) ); const responseData = { categories, count: categories.length, timestamp: new Date().toISOString(), }; await cache.set(cacheKey, responseData, 60 * 60); return NextResponse.json(responseData); } catch (error) { console.error("Error fetching categories:", error); return NextResponse.json( { error: "Failed to fetch categories" }, { status: 500 } ); } } Next, we create a folder in the api directory called cache and in it, a sub-folder called invalidate and a route.ts file. TypeScript import { NextRequest, NextResponse } from "next/server"; import { getCacheInstance } from "@/lib/redis"; export async function POST(request: NextRequest) { try { const { pattern } = await request.json(); if (!pattern) { return NextResponse.json( { error: "Pattern is required" }, { status: 400 } ); } const cache = getCacheInstance("api:"); const deletedCount = await cache.invalidatePattern(pattern); return NextResponse.json({ success: true, message: `Invalidated ${deletedCount} cache entries`, pattern, }); } catch (error) { console.error("Error invalidating cache:", error); return NextResponse.json( { error: "Failed to invalidate cache" }, { status: 500 } ); } } Step 5: UI — Building the Rate Limiting Demo UI In the app directory, create a folder called demo and two sub-folders called caching and rate-limiting respectively. In the rate-limiting sub-folder, create a page.tsx file and paste below. TypeScript "use client"; import { useState } from "react"; import Link from "next/link"; export default function RateLimitingDemo() { const [requests, setRequests] = useState<any[]>([]); const [loading, setLoading] = useState(false); const [requestCount, setRequestCount] = useState(5); const [endpoint, setEndpoint] = useState("/api/tests/basic"); async function makeRequests() { setLoading(true); setRequests([]); const newRequests = []; const promises = Array.from({ length: requestCount }, (_, i) => { return fetch(endpoint) .then((response) => { const headers = { status: response.status, limit: response.headers.get("X-RateLimit-Limit"), remaining: response.headers.get("X-RateLimit-Remaining"), reset: response.headers.get("X-RateLimit-Reset"), }; return response .json() .then((data) => { return { id: i + 1, headers, data, success: response.ok }; }) .catch(() => { return { id: i + 1, headers, data: { error: "Failed to parse JSON" }, success: false, }; }); }) .catch((error) => { return { id: i + 1, headers: {}, data: { error: error.message }, success: false, }; }); }); const results = await Promise.all(promises); setRequests(results); setLoading(false); } return ( <div className="min-h-screen p-8 bg-black text-black "> <div className="max-w-5xl mx-auto"> <div className="mb-6"> <Link href="/" className="text-blue-600 hover:underline"> ← Back to home </Link> </div> <h1 className="text-3xl font-bold mb-6">Redis Rate Limiting Demo</h1> <div className="bg-white p-6 rounded-lg shadow-md mb-8"> <h2 className="text-xl font-semibold mb-4">How It Works</h2> <p className="mb-4"> This demo shows Redis-based rate limiting in action. We've configured the API to limit requests based on client IP address. </p> <p> Try making multiple requests at once to see how the rate limiter kicks in after exceeding the allowed limit. </p> </div> <div className="bg-white p-6 rounded-lg shadow-md mb-8"> <h2 className="text-xl font-semibold mb-4">Test Rate Limiting</h2> <div className="flex flex-col md:flex-row gap-4 mb-6"> <div className="flex-1"> <label htmlFor="requestCount" className="block text-sm font-medium text-gray-700 mb-1" > Number of Requests </label> <input type="number" id="requestCount" min="1" max="20" value={requestCount} onChange={(e) => setRequestCount(parseInt(e.target.value))} className="w-full p-2 border border-gray-300 rounded-md" /> </div> <div className="flex-1"> <label htmlFor="endpoint" className="block text-sm font-medium text-gray-700 mb-1" > API Endpoint </label> <select id="endpoint" value={endpoint} onChange={(e) => setEndpoint(e.target.value)} className="w-full p-2 border border-gray-300 rounded-md" > <option value="/api/tests/basic"> Default Rate Limit (60/min) </option> <option value="/api/tests/auth"> Auth Rate Limit (10/min) </option> <option value="/api/tests/admin"> Admin Rate Limit (3/min) </option> </select> </div> <div className="flex-1 flex items-end"> <button onClick={makeRequests} disabled={loading} className="w-full bg-blue-600 text-white py-2 px-4 rounded hover:bg-blue-700 disabled:bg-blue-300" > {loading ? "Sending Requests..." : "Send Requests"} </button> </div> </div> </div> {requests.length > 0 && ( <div className="bg-white p-6 rounded-lg shadow-md"> <h2 className="text-xl font-semibold mb-4">Request Results</h2> <div className="overflow-x-auto"> <table className="min-w-full divide-y divide-gray-200"> <thead className="bg-gray-50"> <tr> <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider"> # </th> <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider"> Status </th> <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider"> Rate Limit </th> <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider"> Remaining </th> <th className="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider"> Response </th> </tr> </thead> <tbody className="bg-white divide-y divide-gray-200"> {requests.map((request) => ( <tr key={request.id}> <td className="px-6 py-4 whitespace-nowrap"> {request.id} </td> <td className="px-6 py-4 whitespace-nowrap"> <span className={`px-2 inline-flex text-xs leading-5 font-semibold rounded-full ${ request.success ? "bg-green-100 text-green-800" : "bg-red-100 text-red-800" }`} > {request.headers.status || "Error"} </span> </td> <td className="px-6 py-4 whitespace-nowrap"> {request.headers.limit || "-"} </td> <td className="px-6 py-4 whitespace-nowrap"> {request.headers.remaining || "-"} </td> <td className="px-6 py-4"> <pre className="text-xs bg-gray-50 p-2 rounded overflow-x-auto"> {JSON.stringify(request.data, null, 2)} </pre> </td> </tr> ))} </tbody> </table> </div> </div> )} </div> </div> ); } Step 6: UI — Building the Caching Demo UI In the demo/caching folder, create a page.tsx file and copy the code below. TypeScript "use client"; import { useState, useEffect } from "react"; import Link from "next/link"; import { Product } from "@/lib/product"; export default function CachingDemo() { const [products, setProducts] = useState<Product[]>([]); const [loading, setLoading] = useState(false); const [responseTime, setResponseTime] = useState(0); const [cacheStatus, setCacheStatus] = useState(""); const [category, setCategory] = useState(""); const [categories, setCategories] = useState<string[]>([]); useEffect(() => { async function fetchCategories() { const response = await fetch("/api/products/categories"); const data = await response.json(); setCategories(data.categories); } fetchCategories(); }, []); async function fetchProducts() { setLoading(true); setCacheStatus(""); const startTime = performance.now(); let url = "/api/products"; if (category) { url += `?category=${category}`; } const response = await fetch(url); const data = await response.json(); const endTime = performance.now(); setProducts(data.products); setResponseTime(endTime - startTime); setCacheStatus(endTime - startTime < 500 ? "HIT" : "MISS"); setLoading(false); } async function invalidateCache() { setLoading(true); const response = await fetch("/api/cache/invalidate", { method: "POST", headers: { "Content-Type": "application/json", }, body: JSON.stringify({ pattern: "products", }), }); const data = await response.json(); alert(`Cache invalidated: ${data.message}`); setLoading(false); fetchProducts(); } return ( <div className="min-h-screen p-8 bg-black text-black"> <div className="max-w-5xl mx-auto"> <div className="mb-6"> <Link href="/" className="text-blue-600 hover:underline"> ← Back to home </Link> </div> <h1 className="text-3xl font-bold mb-6">Redis Caching Demo</h1> <div className="bg-white p-6 rounded-lg shadow-md mb-8"> <h2 className="text-xl font-semibold mb-4">How It Works</h2> <p className="mb-4"> This demo shows Redis caching in action. When you fetch products, the first request will be slow as it goes to the database. Subsequent requests will be fast as they come from the Redis cache. </p> <p> Try fetching products multiple times to see the difference in response time between cache misses and hits. </p> </div> <div className="bg-white p-6 rounded-lg shadow-md mb-8"> <h2 className="text-xl font-semibold mb-4">Test Caching</h2> <div className="flex flex-col md:flex-row gap-4 mb-6"> <div className="flex-1"> <label htmlFor="category" className="block text-sm font-medium text-gray-700 mb-1" > Filter by Category </label> <select id="category" value={category} onChange={(e) => setCategory(e.target.value)} className="w-full p-2 border border-gray-300 rounded-md" > <option value="">All Categories</option> {categories.map((cat) => ( <option key={cat} value={cat}> {cat.charAt(0).toUpperCase() + cat.slice(1)} </option> ))} </select> </div> <div className="flex-1 flex items-end"> <button onClick={fetchProducts} disabled={loading} className="w-full bg-blue-600 text-white py-2 px-4 rounded hover:bg-blue-700 disabled:bg-blue-300" > {loading ? "Loading..." : "Fetch Products"} </button> </div> <div className="flex-1 flex items-end"> <button onClick={invalidateCache} disabled={loading} className="w-full bg-red-600 text-white py-2 px-4 rounded hover:bg-red-700 disabled:bg-red-300" > Invalidate Cache </button> </div> </div> {responseTime > 0 && ( <div className="mb-4 p-4 bg-gray-100 rounded-md"> <div className="flex justify-between items-center"> <p className="text-lg"> Response Time:{" "} <span className="font-bold">{responseTime.toFixed(2)}ms</span> </p> <p className="text-lg"> Cache:{" "} <span className={`font-bold ${ cacheStatus === "HIT" ? "text-green-600" : "text-yellow-600" }`} > {cacheStatus} </span> </p> </div> </div> )} </div> {products.length > 0 && ( <div className="bg-white p-6 rounded-lg shadow-md"> <h2 className="text-xl font-semibold mb-4"> Products ({products.length}) </h2> <div className="grid md:grid-cols-2 lg:grid-cols-3 gap-4"> {products.map((product) => ( <div key={product.id} className="border rounded-md p-4"> <h3 className="font-medium text-lg mb-1">{product.name}</h3> <p className="text-gray-500 text-sm mb-2"> {product.category} </p> <p className="font-bold text-lg mb-2"> ${product.price.toFixed(2)} </p> <p className="text-sm text-gray-700">{product.description}</p> </div> ))} </div> </div> )} </div> </div> ); } Testing Our Application Phew! That was a lot of code, but finally we can test it out. The first thing is to run our development server. Plain Text npm run dev To test the features: 1. Caching Demo Click on the caching demo.Select a category and click "Fetch Products."Note the response time.Click "Fetch Products" again and observe the much faster response time, as well as cache hits and misses.Try clicking "Invalidate Cache" and then "Fetch Products" again. Initial response time is 2004.50 ms (without cache). Response time with cache — 285.50ms! 2. Rate-Limiting Demo Click on the rate-limiting demo.Select the "Admin Rate Limit" endpoint.Set the number of requests to five.Click "Send Requests."Observe how the first three requests succeed, but the next two fail with a 429 error.Try different endpoints with different rate limits. The admin rate limit allows just one request per minute. You can see the first three requests were successful, and the other two failed. When we make a request again within the same minute, we see that all request fails. How It All Works Let's review how the different components of our application interact: Caching Flow The user selects a category and clicks "Fetch Products" in the UI.The client sends a request to /api/products?category=xyz.The API route checks Redis for a cached response using getCacheInstance().If a cache hit occurs: The cached data is returned with an X-Cache-Status: HIT headerThe response is very fast (usually under 50ms)If a cache miss occurs: The data is fetched from the product serviceThe data is stored in Redis with a 5-minute TTLThe response is returned (slower, typically 300–1000ms)The UI displays the products and shows the response time Rate Limiting Flow The user selects an endpoint and number of requests, then clicks "Send Requests."The client sends multiple requests to the selected endpointFor each request, the middleware intercepts the request and calls the appropriate rate limiterThe rate limiter: Tracks the request count for the client's IP addressAdds rate limit headers to the responseIf the limit is exceeded, it returns a 429 errorThe UI displays the results, showing status codes, remaining requests, and response data Conclusion This is a demonstration on how to implement caching and rate-limiting using Redis. It is a long tutorial, but the intention is to be as elaborate as possible, so it's easy to understand. You can also find the complete project here. Thanks for reading! More

Why Documentation Matters More Than You Think

By Otavio Santana

CORE

Software engineers love to ship. There’s something exhilarating about writing code, pushing it to production, and watching it solve real-world problems. But in the rush to build, we often overlook one of the most powerful tools in our engineering toolbox: documentation. Documentation isn’t just about writing things down. It’s about building a system that others can understand, improve, and trust — even when the original developers have moved on. And when documentation is missing or poorly maintained, the consequences can be devastating. Good documentation doesn’t just sit on the sidelines — it actively accelerates development. When well-maintained, it becomes a silent team member that answers questions, clarifies decisions, and reduces uncertainty. It speeds up onboarding by providing newcomers with a clear path to understanding how the system works, eliminating the need for hours of one-on-one guidance. It also decreases the number of unnecessary meetings and interruptions by making key knowledge easily accessible to everyone, asynchronously. Moreover, documentation brings clarity to complexity. It simplifies software development by breaking down how things work, why decisions were made, and how components fit together. This leads to better design, fewer bugs, and faster iterations, because the entire team is aligned and empowered to move forward with confidence. There are real-world cases where the absence of proper documentation contributed to multimillion-dollar losses — and even total business failure. Let’s explore why documentation is crucial, the various types we rely on, and how it ultimately serves as the foundation for sustainable, scalable software systems. When Documentation Is Missing: The Knight Capital Disaster One of the most infamous examples of what happens when documentation is neglected is the Knight Capital Group incident. In 2012, this high-frequency trading firm lost $440 million in just 45 minutes. The cause? A software deployment gone wrong. A configuration flag that had been repurposed wasn’t documented. During deployment, some servers inadvertently activated outdated code. This legacy code started executing thousands of erroneous trades. The result was catastrophic and irreversible. Although several factors were at play — poor testing and incomplete rollout processes — one glaring issue was the lack of operational documentation. Engineers didn’t fully understand how all parts of the system interacted. The decision to skip documentation had existential consequences: Knight Capital filed for bankruptcy shortly after. This incident serves as a harsh reminder: undocumented systems are inherently dangerous Two Sides of the Same Coin: Tactical and Strategic Documentation To navigate the complexity of modern software, documentation usually falls into two categories: Tactical documentation, which lives close to the codebase and supports daily development tasks.Strategic documentation provides a comprehensive overview of the architecture, key decisions, and the system’s evolution. The documentation samples. Think of tactical documentation as your "compass" — helping you orient yourself quickly — and strategic documentation as your "map" — guiding you through the terrain. You need both to travel effectively. Tactical Documentation: Supporting Daily Development Tactical documentation is what developers use the most, because it’s right where the code is. It explains how to run, build, test, and contribute to a project. It’s the first stop when you join a new team or revisit a project after a few months. README Files A well-written README answers the most basic — but essential — questions: What is this? How do I run it? How do I contribute? Trade-off: When neglected, READMEs quickly become outdated or incomplete. Keep it concise and focused on what a developer needs on their first day. Changelog A changelog documents what changed between versions—new features, fixes, deprecations. It promotes transparency, enabling users to track their progress. Trade-off: If not curated carefully, changelogs can become noise. Avoid dumping auto-generated commit logs; curate them for clarity. JavaDoc (or Equivalents) Inline documentation that describes the purpose and usage of classes and methods. JavaDoc can act as a live contract between components or teams. Trade-off: Inline comments often fall behind the code. Keep them focused on why something exists rather than what — the latter should be evident from the code itself. Strategic Documentation: Guiding Long-Term Decisions While tactical documentation helps you work in the system, strategic documentation enables you to understand the system as a whole. It answers the big questions: Why was this decision made? How does the system fit into the larger architecture? C4 Model The C4 model offers a layered approach to visualizing software architecture. It starts with high-level system context and drills down to containers, components, and code. Why it matters: It fosters a shared understanding among both technical and non-technical stakeholders. Diagrams aren’t just decoration — they become a decision-making tool. Trade-off: Keeping diagrams up to date requires discipline and tooling. Automated code generation, such as using Structurizr, can be helpful. ADRs (Architecture Decision Records) ADRs are short documents that capture a single architectural decision. They encompass the context, decision-making process, reasoning, and consequences. Why it matters: They act as a time capsule. Months or years later, you’ll understand why a path was chosen — and whether it’s still valid. Trade-off: Without structure, ADRs can become cluttered. Use consistent naming and templates. Tech Radar A Tech Radar visualizes the maturity and adoption of technologies across your organization. It helps standardize tools and highlight emerging practices. Why it matters: Teams align better when they know which technologies are encouraged and which are being phased out. Trade-off: Requires regular review to maintain relevance. Without ownership, it becomes obsolete. The Rise of DocOps: Documentation as a First-Class Citizen In recent years, teams have begun to adopt DocOps — short for Documentation Operations — as a formal approach to integrating documentation into their development workflows. DocOps treats documentation like code. That means: It lives in the same repository.It uses version control (Git).It follows a review process through pull requests.It gets built, tested, and deployed just like software. This approach ensures that documentation evolves in lockstep with the code. Developers don’t have to dig through wikis or stale Confluence pages; everything they need is right there in the project. Two popular formats support this workflow: Markdown Lightweight and widely supported, Markdown is ideal for README files, changelogs, and documentation portals on GitHub or GitLab. It’s readable both in plain text and rendered format. AsciiDoc A more powerful alternative to Markdown, AsciiDoc supports advanced formatting, includes, variables, and styling. It's often used for larger, structured documentation projects and is compatible with tools like Antora. The key benefit? Documentation close to the code is easier to maintain. When engineers change code, they can update the relevant docs in the same commit. The result is fresher, more reliable documentation that evolves with your system. Documentation Is an Engineering Practice Just as writing tests, setting up CI/CD, or tracking technical debt are core parts of engineering hygiene, writing and maintaining documentation is also a crucial aspect. It prevents knowledge silos, enables team autonomy, and reduces onboarding time. Moreover, it fosters clarity of thought. The act of documenting often uncovers design flaws or edge cases you hadn’t considered. And if you’ve ever told yourself, “I’ll remember why I did this,” chances are you won’t. Documentation is your backup memory. It’s not for today — you write it so someone else (including future-you) can succeed tomorrow. Final Thoughts: From Chaos to Clarity The best software teams don’t just build things quickly — they make things that last. And lasting systems need more than elegant code. They need clarity, context, and continuity. That’s what documentation provides. Whether it's a simple Markdown README, a C4 architecture diagram, or a series of ADRs, good documentation ensures that everyone can contribute with confidence and navigate complexity with ease. Ultimately, documentation isn’t an afterthought. It’s a form of engineering. And the cost of skipping it—ask Knight Capital — can be far greater than we imagine. Video More

Mastering Fluent Bit: Installing and Configuring Fluent Bit Using Container Images (Part 2)

By Eric D. Schabell

CORE

Simplifying Vector Embeddings With Go, Cosmos DB, and OpenAI

By Abhishek Gupta

CORE

Thumbnail Generator Microservice for PDF in Spring Boot

By Amol Gote

CORE

Security in the CI/CD Pipeline

As development cycles speed up, integrating security into the CI/CD pipeline is essential to embed security checks early and throughout the process to prevent vulnerabilities from creeping into production. This article delves deeper into the technical details, code examples, and configurations to build a robust, secure CI/CD pipeline. A CI/CD pipeline is an automated workflow that streamlines the process of software development from code integration to deployment. It comprises two main practices: Continuous integration (CI): Developers frequently merge their code changes into a shared repository. Each change triggers automated builds and tests to catch issues early.Continuous delivery/deployment (CD): After the CI phase, the software is automatically prepared for release. In continuous delivery, manual approval is usually needed before deployment, while continuous deployment pushes every change to production automatically. By automating these processes, a CI/CD pipeline improves code quality, accelerates release cycles, and reduces human error. Why Secure the CI/CD Pipeline? Modern CI/CD pipelines are complex systems that automate the building, testing, and deployment of applications. However, their complexity also increases the potential attack surface. Integrating security early and continuously (a practice known as "shift left" security) enables teams to: Detect vulnerabilities early. Automated security tests catch issues during development rather than in production.Comply with regulatory standards. Incorporate security controls and documentation to meet compliance requirements.Maintain developer velocity. Automated security feedback reduces manual checks, allowing teams to focus on writing code. Security Stages 1. Source Code Analysis and Static Application Security Testing (SAST) Static Analysis Tools and Integration Static code analysis tools such as SonarQube, ESLint (for JavaScript), and FindBugs (for Java) analyze your source code for vulnerabilities, code smells, and security anti-patterns. For example, SonarQube not only reports code quality issues but also enforces quality gates to fail builds if security issues are detected. Example: Jenkins Integration with SonarQube Groovy pipeline { agent any environment { SONARQUBE_SERVER = 'SonarQubeServer' } stages { stage('Build') { steps { echo 'Building the application...' sh './build.sh' } } stage('Static Code Analysis') { steps { echo 'Running SonarQube analysis...' withSonarQubeEnv(SONARQUBE_SERVER) { sh 'sonar-scanner -Dsonar.projectKey=MyApp -Dsonar.sources=./src -Dsonar.host.url=$SONAR_HOST_URL' } } } } post { failure { echo 'Build failed due to code quality issues. Check SonarQube for details.' } } } 2. Dependency and Library Scanning Open-source dependencies are a common source of vulnerabilities. Tools such as OWASP Dependency-Check, Snyk, and Black Duck scan your project’s dependencies for known vulnerabilities using CVE databases. Groovy pipeline { agent any stages { stage('Build') { steps { echo 'Building the application...' sh './build.sh' } } stage('Dependency Scan') { steps { echo 'Running dependency scan with OWASP Dependency-Check...' // The command scans the current directory and generates an HTML report. sh 'dependency-check.sh --project MyApp --scan . --format "HTML" --out dependency-report.html' } post { always { // Archive the generated report for later review. archiveArtifacts artifacts: 'dependency-report.html', allowEmptyArchive: true echo 'Dependency scan completed and report archived.' } failure { echo 'Dependency scan encountered issues. Please check the logs and the report for details.' } } } stage('Test') { steps { echo 'Running tests...' sh './run_tests.sh' } } } } 3. Container Security Scanning Container images can harbor vulnerabilities if not properly scanned. Tools such as Trivy inspect container images for high-severity issues. Groovy pipeline { agent any environment { DOCKER_IMAGE = "my-app:${env.BUILD_ID}" } stages { stage('Build Container Image') { steps { echo 'Building Docker image...' sh "docker build -t ${DOCKER_IMAGE} ." } } stage('Container Scan') { steps { echo 'Scanning container image for vulnerabilities using Trivy...' // Fail the stage if high or critical vulnerabilities are found. sh "trivy image --exit-code 1 --severity HIGH,CRITICAL ${DOCKER_IMAGE}" } } } } 4. Infrastructure as Code (IaC) Scanning For organizations that use IaC (e.g., Terraform or CloudFormation), tools like Checkov ensure that your infrastructure code complies with security best practices. Groovy pipeline { agent any stages { stage('IaC Scan') { steps { echo 'Scanning Infrastructure as Code with Checkov...' // Assume Terraform code is located in the 'terraform' directory. sh 'checkov -d ./terraform/' } } } } 5. Dynamic Application Security Testing (DAST) DAST tools (such as OWASP ZAP) simulate attacks against a running application to find vulnerabilities that occur at runtime. Groovy pipeline { agent any stages { stage('Deploy to Test Environment') { steps { echo 'Deploying to test environment...' sh './deploy_test.sh' } } stage('Dynamic Application Security Testing (DAST)') { steps { echo 'Running OWASP ZAP scan...' // Start ZAP in daemon mode and perform a quick scan. sh 'zap.sh -daemon -port 8080 -config api.disablekey=true' // Allow ZAP time to initialize. sleep 10 sh 'zap-cli quick-scan --self-contained --start-options "-config api.disablekey=true" http://test-env-url' } } } } 6. Secrets Management Avoid hard-coding secrets in your code. Instead, use CI/CD environment variables or external secret management tools. Groovy pipeline { agent any environment { API_KEY = credentials('api-key-id') DB_PASSWORD = credentials('db-password-id') } stages { stage('Deploy') { steps { echo 'Deploying to production using secure credentials...' sh "./deploy.sh --api-key '${API_KEY}' --db-password '${DB_PASSWORD}'" } } } } Conclusion Integrating robust security checks into your CI/CD pipeline is a continuous, multi-layered process. You can significantly reduce the risk of vulnerabilities entering production by embedding static analysis, dependency scanning, container and IaC security, dynamic testing, and secrets management into your Jenkins Pipeline using Groovy. As security threats evolve, continuously updating these processes and tools is key to maintaining a strong security posture. Feel free to adapt and expand these examples to fit your environment. Remember, securing your CI/CD pipeline is an ongoing commitment. Happy coding, and stay secure!

By Elakkiya Daivam

Building a Scalable ML Pipeline and API in AWS

With rapid progress in the fields of machine learning (ML) and artificial intelligence (AI), it is important to deploy the AI/ML model efficiently in production environments. This blog post discusses an end-to-end ML pipeline on AWS SageMaker that leverages serverless computing, event-trigger-based data processing, and external API integrations. The architecture downstream ensures scalability, cost efficiency, and real-time access to applications. In this blog, we will walk through the architecture, explain design decisions, and examine the key AWS services used to build this system. Architecture Overview The AWS-based ML pipeline consists of multiple components that communicate with one another seamlessly to perform model execution, data storage, processing, and API exposure. The workflow includes: ML Model Execution in AWS SageMakerStoring data in AWS S3, DynamoDB, and SnowflakeEvent-based processing using AWS Lambda and AWS GlueReal-time API integration with AWS Lambda and Application Load BalancerRouting traffic to applications through AWS Route 53 Step 1: Running of the ML Model on AWS SageMaker The main component of the system is the ML model that runs on AWS SageMaker periodically to generate predictions. This is also called batch processing. The SageMaker pipeline: Uses preprocessed data and results from previous runs.Applies the ML algorithms for inference.Writes the output in both JSON and Delta formats to an S3 bucket. Why save data in JSON and Delta formats? JSON is lightweight and can be easily consumed by AWS DynamoDB for real-time querying.Delta format allows for efficient data loading into Snowflake for analytics and reporting. Step 2: Event-Based Data Processing and Storage Once SageMaker writes the output to an S3 bucket, an event-based trigger will automatically run the next steps. S3 Event Notification invokes an AWS Lambda function, as soon as the new "done" file is created in the corresponding S3 location, where the trigger was setup.The Lambda function invokes the AWS Glue job that: Processes and loads the JSON data from the S3 location into DynamoDB.Copies Delta data to Snowflake. Why use AWS Glue for data ingestion? AWS Lambda has a max timeout of 15 minutes.Processing and uploading huge amounts of data might take more than 15 minutes.Glue ETL transformations ensure that structured and clean data ingestion is assured. Step 3: API Processing and Real-Time Access Now, the data stored in DynamoDB needs to be accessed by external applications. That’s done using APIs. We can use an AWS Lambda function to host the API code. API Lambda function is invoked when the application makes a request.The API Lambda function: Queries DynamoDB with the latest ML model results.Integrates with real-time APIs (third-party services) to enhance the results.Processes all this information and generates an API response. Step 4: API Exposure Using Application Load Balancer (ALB) To handle API traffic, the Lambda function is connected to an AWS Application Load Balancer (ALB). Why use an Application Load Balancer? ALB routes traffic to the relevant Lambda function.Autoscales based on the number of API requests, ensuring high availability.Distributes traffic efficiently across multiple Lambda instances.Secures the API endpoints by performing authentication and request filtering. Step 5: Routing API Calls Using Route 53 We integrate AWS Route 53 with the ALB to obtain a consistent API endpoint. Route 53 handles domain name resolution, making sure that the applications can easily connect to the API.It also supports custom domain mapping, allowing other teams to use a user-friendly API URL instead of directly accessing ALB endpoints.If the API Lambda is deployed in multiple regions, Route 53 can be configured to route traffic efficiently, ensuring reliability and failover even during high-traffic periods. Most Critical Features of This Architecture Scalability – AWS services like SageMaker, Lambda, Glue, and DynamoDB handle loads dynamicallyCost optimization – Use of on-demand DynamoDB, serverless Lambda, and event-based processing ensures efficient utilization of resourcesReal-time processing – Provides real-time access to the ML output with low-latency APIsSeamless integration – Supports integration with other real-time APIs, thereby enhancing resultsCross-team collaboration – Exporting data to Snowflake helps businesses and other teams to run analytics against ML predictions Future Enhancements and Considerations Streaming processing – Replacing batch flows with Kafka or Kinesis for real-time data processing.Automated model retraining – Use SageMaker Pipelines for automated model retraining. Conclusion This AWS-based ML architecture provides a scalable, automated, and efficient pipeline for running ML models, generating predictions, and serving real-time API responses. By utilizing AWS services such as SageMaker, Lambda, Glue, DynamoDB, ALB, and Route 53, the system ensures cost efficiency, high performance, and real-time data availability for downstream applications. Would love to hear your thoughts!

By Dhanush Thirugnana

Self-Healing Data Pipelines: The Next Big Thing in Data Engineering?

I'm an enthusiastic data engineer who always looks out for various challenging problems and tries to solve them with a simple POC that everyone can relate to. Recently, I have thought about an issue that most data engineers face daily. I have set alerts on all the batch and streaming data pipelines. When the errors reach a threshold limit or if the data pipeline fails, we get failure notifications immediately in the email inbox. Everything seemed fine until I noticed one of our critical datasets could not be loaded into BigQuery. After investigating the error logs, I found several messages with "missing required data." I felt ‘lost’ seeing these frequent raw data issues from a user-inputted file. Handling data inconsistencies, notably missing or formatting data issues, can cause significant problems downstream in analytics and operational workflows. There is a downstream report that relies on this ingested data. This report is extensively used for daily reporting on the key metrics of how business reflects in several areas and gives crucial data points to decide how to navigate. In this critical report, all C-level stakeholders rely on presenting metrics and discussing challenges and the go-forward plan. I spent hours manually inspecting the source CSV files, as this vast file consists of an enormous volume of transactions sourced from another upstream application. Identifying the problematic rows and correcting them is vital. By the time I solved the issues, the deadline had passed, and the stakeholders were understandably frustrated. That day, I realized how fragile traditional data pipelines can be. They break easily, and fixing them multiple times requires manual intervention, which is time-consuming and error-prone. Have you ever faced a similar situation? Have you spent countless hours debugging data pipelines, only to realize the root cause was a simple formatting error or missing required field? If so, you're not alone. Data engineers worldwide struggle with these challenges daily. But what if there was a way to build pipelines that could "heal" themselves? That's precisely what I set out to achieve. How Self-Healing Data Pipeline Works The idea of a self-healing pipeline is simple: When errors occur during data processing, the pipeline should automatically detect, analyze, and correct them without human intervention. Traditionally, fixing these issues requires manual intervention, which is time-consuming and prone to errors. There are several ways to idealize this, but using AI agents is the best method and a futuristic approach for data engineers to self-heal failed pipelines and auto-correct them dynamically. In this article, I will show a basic implementation of how to use LLMs like the GPT-4/DeepSeek R1 model to self-heal data pipelines by using LLM’s recommendation on failed records and applying the fix through the pipeline while it is still running. The provided solution can be scaled to large data pipelines and extended to more functionalities by using the proposed method. Here's how I built a practical pipeline leveraging OpenAI API that uses the GPT-4 model in a cloud environment. The basic steps followed are: Upload a source file into Google Cloud Storage Bucket. You can use any local or any other cloud storage if you don't have access to the GCP cloud.Create the data model for ingesting the original data into the BigQuery table and error records into the Error table.Read the source file from the CSV and identify the Clean dataset and invalid records (error rows) dataset from the input data.Ingest the Clean dataset into BigQuery and pass the Error Rows dataset to LLM using the Prompt.For each Error Row, OpenAI’s GPT API analyzes and provides an intelligent product ID assignment.Google BigQuery to store and retrieve product information dynamically.Python-based automation for seamless integration. Here is the complete codebase on my GitHub. 1. Reading Input Data from Cloud Storage The pipeline starts by reading a CSV file uploaded by clients stored in Cloud Storage. We can leverage a Cloud Function — serverless execution of pipeline steps — to trigger whenever a new file is uploaded to the bucket. The function reads the file using the google-cloud-storage library and parses it into a Pandas DataFrame for further processing. You can write some data quality checks before passing on to the next step, where the data gets ingested into the BigQuery table. However, data issues are dynamic in real-world scenarios, and you can't predict and write all test cases, which makes the code complex and unreadable. In our use case, the CSV file contains fields ProductID, Price, name, saleAmount. Below is the sample file with data (also missing data in both ProductID and Price fields). Python # Read CSV from GCS client = storage.Client() bucket = client.bucket(bucket_name) blob = bucket.blob(file_name) data = blob.download_as_text() df = pd.read_csv(io.StringIO(data)) 2. Ingesting Data into BigQuery Next, the pipeline attempts to load the data into BigQuery. If any rows fail due to schema mismatches, data type errors, or missing fields, they are captured and logged for further analysis. This step is essential to detect the underlying error information that will be used to identify a possible solution by OpenAI API. Python # Function to clean and preprocess data def clean_data(df): avg_price = get_average_price() df["Price"] = df["Price"].fillna(avg_price) # Log and remove rows with issues error_rows = df[df["ProductID"].isna()] clean_df = df.dropna(subset=["ProductID"]) return clean_df, error_rows # Function to query BigQuery for an average price def get_average_price(): client = bigquery.Client() query = f"SELECT AVG(Price) AS avg_price FROM `{BQ_PROJECT_ID}.{BQ_DATASET_ID}.Product_Info`" try: df = client.query(query).to_dataframe() avg_price = df["avg_price"][0] print(f"Fetched Average Price: {avg_price}") return avg_price except Exception as e: print(f"Error fetching average price: {e}") return None Notice that avg_price = get_average_price() is getting it from BigQuery lookup. After the clean dataset insert: 3. Analyzing Errors With LLMs Analyzing errors is the key step in the entire pipeline, and this is where the magic happens. The failed records were sent to a large language model (LLM) like GPT-4 or DeepSeek R1 for analysis. The LLM examines the errors and suggests corrections and a fixed record. For example, suppose a date field is formatted incorrectly. In that case, the LLM might suggest the correct formatted record from a String to an Integer conversion or String to Date/Timestamp conversion and vice versa. In cases where the data is expected but found null, based on the rules enforced by our code, missing values with 'Average' or Default values are fixed to ensure data ingestion is successful. Implementing the ChatCompletion Request With Retry Mechanism To ensure resilience, we implement a retry mechanism using tenacity. The function sends error details to GPT and retrieves suggested fixes. In our case, the 'functions' list was created and passed to the JSON payload using the ChatCompletion Request. Note that the 'functions' list is the list of all functions available to fix the known or possible issues using the Python functions we have created in our pipeline code. GPT analyzes the input prompt and the error message to determine if it is best to invoke a specific function listed on the 'functions' list to fix the issue. Accordingly, GPT’s response provides structured data indicating which function should be called. GPT does not execute the function directly. Instead, the pipeline will execute it. Python @retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3)) def chat_completion_request(messages, functions=None, model=GPT_MODEL): headers = { "Content-Type": "application/json", "Authorization": "Bearer " + openai.api_key, } json_data = {"model": model, "messages": messages} if functions is not None: json_data.update({"functions": functions}) try: response = requests.post( "https://api.openai.com/v1/chat/completions", headers=headers, json=json_data, ) return response.json() except Exception as e: print("Unable to generate ChatCompletion response") print(f"Exception: {e}") return e # Function to send ChatCompletion request to OpenAI API functions = [ { "name": "assign_product_id", "description": "assigning a unique ProductID", "parameters": { "type": "object", "properties": { "ProductID": { "type": "integer", "description": "The product ID to assign." }, } }, } ] assign_product_id is the function listed on the 'functions' list that GPT can call if required. In our case, the ProductID is missing for the last two rows in the CSV file. Hence, the GPT called the specific assign_product_id function to determine the ProductID value. The function assign_product_id fetches the highest assigned ProductID from BigQuery and increments it for subsequent use. If it runs for the first time or no data is available in the BigQuery table, it will assign the default 99999 as ProductID. Python def assign_product_id(): client = bigquery.Client() # table_ref = client.dataset(BQ_DATASET_ID).table(BQ_TABLE_ID) query = f""" Select max(ProductID) as max_id from `{BQ_PROJECT_ID}.{BQ_DATASET_ID}.{BQ_TABLE_ID}` WHERE ProductID < 99999 """ df = client try: df = client.query(query).to_dataframe() except Exception as e: print(f"Error fetching max ProductID: {e}") return None return df["max_id"][0] + 1 if not df.empty else 99999 4. Applying Automatic Corrections The pipeline applies the GPT’s suggestions to the failed records and re-attempts to ingest them into BigQuery. The data is stored in the main table if the corrections are successful. If not, the irreparable records are logged into a separate error table for manual review. In instances where the field is required and unique, GPT can get the unique ProductID value from BigQuery and add a plus 1 to it to maintain the uniqueness. Consider cases where there were multiple error rows in the pipeline; each record is processed sequentially with the fix offered by the GPT’s response and updates the Error Record with SUGGESTED values. In the below code, ProductID gets replaced by the value fetched from assign_product_id() BigQuery table. When there are multiple error rows, each receives a unique number by incrementing the ProductID. Python # Function to send error data to GPT-4 for analysis def analyze_errors_with_gpt(error_rows): if error_rows.empty: return error_rows new_product_id = assign_product_id() for index, row in error_rows.iterrows(): prompt = f""" Fix the following ProductID by assigning a unique ProductID from the bigquery table Product_Info: {row.to_json()} """ chat_response = chat_completion_request( model=GPT_MODEL, messages=[{"role": "user", "content": prompt}], functions=functions ) if chat_response is not None: try: if chat_response["choices"][0]["message"]: response_content = chat_response["choices"][0]["message"] else: print("Chat response content is None") continue except json.JSONDecodeError as e: print(f"Error decoding JSON response: {e}") continue if 'function_call' in response_content: if response_content['function_call']['name'] == 'assign_product_id': res = json.loads(response_content['function_call']['arguments']) res['product_id'] = new_product_id error_rows.at[index, "ProductID"] = res['product_id'] new_product_id += 1 # Increment the ProductID for the next row print(f"Assigned ProductID: {res['product_id']}") else: print("Function not supported") else: chat.add_prompt('assistant', response_content['content']) else: print("ChatCompletion request failed. Retrying...") return error_rows 5. Ingesting the Fixed Rows into BigQuery Table The main function reads data from Google Cloud Storage (GCS), cleans it, and loads valid data into BigQuery while fixing errors dynamically. Python # Main function to execute the pipeline def main(): bucket_name = "self-healing-91" file_name = "query_results.csv" # Read CSV from GCS client = storage.Client() bucket = client.bucket(bucket_name) blob = bucket.blob(file_name) data = blob.download_as_text() df = pd.read_csv(io.StringIO(data)) # Clean data and identify errors clean_df, error_rows = clean_data(df) # Load valid data into BigQuery load_to_bigquery(clean_df, BQ_TABLE_ID) # Process errors if any if not error_rows.empty: # Analyze errors with GPT-4 error_rows = analyze_errors_with_gpt(error_rows) load_to_bigquery(error_rows, BQ_TABLE_ID) print("Fixed Errors loaded successfully into BigQuery original table.") Post error fix data, particularly check rows from 66-68. After fetching the max value as 10000 ProductID from the BigQuery table, they're incremented by one. Also, the Price field with missing information in the error rows was replaced by Avg(Price) from the BigQuery table. 6. Logging and Monitoring Throughout the process, errors and pipeline activities are logged using Cloud Logging. This ensures that engineers can monitor the pipeline's health and troubleshoot issues. Tools and Technologies Used Here are the key tools and technologies I used to build and test this pipeline: Cloud storage: This is for storing input CSV files.Cloud functions: For serverless execution of pipeline steps.BigQuery: This is used to store cleaned data and error logs.GPT-4/DeepSeek R1: For analyzing and suggesting corrections for failed records.Cloud logging: This is for monitoring and troubleshooting.Cloud composer: This is used to orchestrate the pipeline using Apache Airflow. Challenges Faced 1. LLM Integration Integrating an LLM into the pipeline was tricky. I had to ensure that the API calls were efficient and that the LLM’s responses were accurate. Additionally, there were cost considerations, as LLM APIs can be expensive for large datasets. Just know that you must set up an API key for this service. For example, for OpenAI, you must visit https://platform.openai.com/ to register and generate the new API key and use it in the pipeline when sending the API call with a prompt. 2. Error Handling Designing a robust error-handling mechanism was challenging. I had to account for various errors, from schema mismatches to network issues, and ensure that the pipeline could handle them gracefully. A pipeline could face numerous problems, and all issues can't be solved dynamically. Examples are File is Empty, and BigQuery Table doesn’t exist. 3. Scalability As the volume of data grew, I had to optimize the pipeline for scalability. This involved partitioning data in BigQuery and using Dataflow for large-scale processing. 4. Cost Management While GCP offers powerful tools, they come at a cost. I had to carefully monitor usage and optimize the pipeline to avoid unexpected bills. OpenAI API cost is another factor that you need to monitor carefully. Final Thoughts and Key Notes Building a self-healing pipeline is a game-changer for data engineers. It reduces manual intervention, improves efficiency, and ensures data quality. However, it’s not a silver bullet. While self-healing pipelines can save time, they come with additional costs, such as LLM API fees and increased Cloud Function usage. It’s essential to weigh these costs against the benefits. If you're new to self-healing pipelines, start with a small project — experiment with LLM integration and error handling before scaling up. Regularly monitor your pipeline’s performance and costs. Use tools like Cloud Monitoring and Cloud Logging to identify bottlenecks and optimize accordingly. Finally, work closely with data scientists, analysts, and business stakeholders to understand their needs and ensure the pipeline delivers value when business requirements change. In conclusion, self-healing pipelines represent the future of data engineering. We can build robust, efficient, intelligent pipelines that minimize downtime and maximize productivity by leveraging tools like GCP and LLMs. Please explore this approach if you've ever struggled with fragile data pipelines. The initial effort is worth the long-term benefits. What are your thoughts on self-healing pipelines? Have you tried building one? Share your experiences in the comments below.

By Naresh Erukulla

A Comprehensive Guide to Protect Data, Models, and Users in the GenAI Era

Editor's Note: The following is an article written for and published in DZone's 2025 Trend Report, Generative AI: The Democratization of Intelligent Systems. Generative AI (GenAI) is transforming how organizations operate, enabling automation, content generation, and intelligent decision making at an unprecedented scale. From AI-powered chatbots to advanced code generation and creative design, GenAI is revolutionizing industries by increasing efficiency and innovation. However, alongside these advancements come significant security risks that organizations must address. The challenge is that as AI systems become more intelligent and sophisticated, they also face evolving threats and risks. Ensuring AI security throughout development and deployment is crucial. This article provides practical checklists to help enterprises securely adopt GenAI. By understanding key security risks, implementing essential technologies, and following best practices, organizations can harness the power of GenAI while ensuring their data, models, and users remain protected. The checklists are separated into two categories: Key security risks of GenAIEssential security technologies for GenAI Key Security Risks of Generative AI GenAI introduces new security risks that organizations must address. Threats include data leaks, model manipulation, and unauthorized access. These risks can lead to serious privacy and security breaches without proper safeguards. 1. Data Privacy and Compliance Risks Generative AI can expose sensitive data, leading to legal violations under regulations like GDPR and HIPAA. Organizations face legal, financial, and reputational risks if AI models process confidential information without safeguards. Ensuring compliance requires strict data handling, access controls, and regular audits. For example, in 2023, Samsung employees accidentally leaked confidential company data by entering it into ChatGPT, raising serious concerns about corporate data privacy and AI misuse. Learn more about the accidental data leak here. Here are steps to address data privacy and compliance risks: ☐ Restrict AI access to sensitive data using role-based controls ☐ Implement data anonymization and encryption before AI processing ☐ Audit AI interactions for compliance with GDPR, HIPAA, etc. ☐ Use AI governance tools to enforce data protection policies 2. Misinformation and Bias AI models can generate false or misleading information, commonly called hallucinations. AI may reinforce stereotypes and produce unfair outcomes if trained on biased data. Organizations must ensure that AI-generated content is accurate, ethical, and free from bias. An incident of this nature occurred in 2023 when an AI-powered news website published misleading and fake articles, causing public misinformation and damaging its credibility. To avoid misinformation and bias: ☐ Test AI models regularly for bias and accuracy ☐ Use diverse, high-quality training data ☐ Implement human review for critical AI outputs ☐ Establish AI ethics guidelines to ensure responsible usage 3. Unauthorized Access and Misuse Unauthorized users can access AI models without proper security measures, leading to data theft or manipulation. Both insiders and external hackers pose a risk, especially if API security is weak or misconfigured. In one case, a misconfigured AI chatbot publicly exposed user conversations due to API vulnerabilities, compromising privacy. Here is a checklist to prevent unauthorized access and misuse issues from happening to you: ☐ Enforce multi-factor authentication (MFA) for AI access ☐ Implement role-based access controls ☐ Monitor AI activity logs for suspicious behavior ☐ Conduct regular security audits and penetration tests 4. Data Poisoning Attackers can manipulate AI training data by injecting malicious inputs and corrupting model outputs. This can lead to biased decisions, misinformation, or exploitable vulnerabilities. In one experiment, researchers demonstrated how poisoning AI datasets could manipulate facial recognition systems, causing them to misidentify people. Here is a checklist to prevent data poisoning: ☐ Validate and clean training data before AI processing ☐ Use differential privacy to prevent data manipulation ☐ Deploy anomaly detection tools to identify poisoned data ☐ Retrain models with verified and diverse datasets 5. Fake "ChatGPT" and Impersonation Attacks Fraudsters create fake AI tools mimicking ChatGPT or other AI services to trick users into sharing sensitive data or installing malware. These fake versions often appear as mobile apps, browser extensions, or phishing websites that look nearly identical to real AI platforms. Some have even been found in official app stores, making them seem more trustworthy to unsuspecting users. Once installed, they can steal login credentials and financial information or even spread harmful software across devices. Here is a checklist to prevent fake "ChatGPT" and impersonation attacks: ☐ Use only official AI tools from verified sources ☐ Educate employees on fake AI and phishing scams ☐ Deploy security tools to detect fraudulent AI services ☐ Report fake AI platforms to authorities 6. Model Stealing Attackers can extract proprietary AI models by exploiting APIs and analyzing responses, leading to intellectual property theft and competitive disadvantage. As found in North Carolina State University's research, "Researchers have demonstrated the ability to steal an artificial intelligence (AI) model without hacking into the device where the model was running. The technique is novel in that it works even when the thief has no prior knowledge of the software or architecture that supports the AI." Figure 1. Model-stealing process The diagram illustrates the model-stealing process, where an attacker sends multiple queries to a target machine learning model and collects the corresponding responses. Using these inputs and outputs, the attacker then trains a stolen model that mimics the behavior of the original, potentially leading to intellectual property theft and unauthorized use. To prevent model stealing: ☐ Limit API access and enforce request rate limits ☐ Encrypt AI models during deployment ☐ Use watermarking to track unauthorized usage ☐ Monitor API activity for suspicious extraction patterns 7. Model Inversion Attacks Hackers can reverse-engineer AI models to recover sensitive training data, potentially exposing confidential or personal information. In one instance, researchers reconstructed faces from a facial recognition AI model, revealing private user data used in training. Andre Zhou gathered a list of resources and research related to model inversion attacks in his GitHub Repository. A model inversion attack is similar to a model stealing attack. A model inversion attack extracts sensitive training data by analyzing model outputs, infers private input data, posing a privacy risk, and grants attackers access to confidential or personal data. Meanwhile, a model stealing attack replicates a target model’s functionality using queries and responses, enables intellectual property theft by recreating the model, and allows attackers to obtain a functional copy of the model’s behavior. Here are steps you can take to prevent model inversion attacks: ☐ Use differential privacy to protect training data ☐ Restrict model exposure by limiting API responses ☐ Apply adversarial defenses to prevent inversion attacks ☐ Assess AI models for vulnerabilities regularly 8. AI-Enhanced Social Engineering AI can generate highly realistic phishing emails, deepfake videos, and voice impersonations, making social engineering attacks more effective. For example, cybercriminals used AI-generated voices to impersonate company executives at a European company, successfully authorizing fraudulent financial transactions amounting to €220,000. The following are measures that can be taken to prevent AI-enhanced social engineering: ☐ Train employees to recognize AI-generated scams, using open-source tools like Google's SynthId (or commercial tools) ☐ Deploy AI-powered security tools to detect deepfakes ☐ Use multi-factor authentication for financial transactions ☐ Monitor communications for unusual patterns Essential Security Technologies for GenAI Securing generative AI means using encryption, access controls, and safe APIs. Monitoring tools catch unusual activity, and defenses protect against attacks. Following privacy rules helps keep AI use safe and fair. We also need to consider the following topics to improve the security level when utilizing AI. 1. Data Loss Prevention Data loss prevention (DLP) solutions monitor and control data flow to prevent sensitive information from being leaked or misused. Here are some ways to incorporate DLP solutions: ☐ Use AI-driven DLP tools to detect and block unauthorized data sharing ☐ Apply strict data classification and access policies ☐ Monitor AI-generated outputs to prevent unintentional data leaks ☐ Regularly audit logs for suspicious activity 2. Zero-Trust Architecture Zero-trust architecture (ZTA) enforces strict access controls, verifying every request based on identity, context, and least privilege principles. Here is a checklist to implement zero-trust architecture: ☐ Implement MFA for AI access ☐ Use identity and access management tools to enforce the least privilege ☐ Continuously monitor and verify user and AI interactions ☐ Segment networks to limit AI system exposure You can find a detailed guide about zero-trust architecture here. Figure 2. Zero-trust architecture 3. Encryption and Confidential Computing Encryption secures AI data at rest and in transit, while confidential computing protects sensitive AI operations in secure environments. Here is a checklist to implement encryption and confidential computing: ☐ Encrypt data using AES-256 for storage and TLS 1.2+ for transmission ☐ Use hardware-based secure enclaves for AI processing ☐ Implement homomorphic encryption for privacy-preserving AI computations ☐ Regularly update cryptographic protocols to prevent vulnerabilities Conclusion Securing generative AI means taking the proper steps to protect data, models, and users; therefore, organizations must continuously improve their security strategies and proactively address key security risks. This can be done in part by incorporating strong access controls, data protection policies, and regular security tests, and doing the proper research to ensure organizations are meeting their own needs as well as regulatory requirements. By following the checklists presented in this article, organizations can safely and innovatively use generative AI. References: "Fake ChatGPT apps spread Windows and Android malware" by Graham Cluley"DeepSeek Data Leak Exposes 1 Million Sensitive Records" by Lars Daniel"Samsung Bans ChatGPT Among Employees After Sensitive Code Leak" by Siladitya Ray"Face Reconstruction from Face Embeddings using Adapter to a Face Foundation Model" by Hatef Otroshi Shahreza, et al."Researchers Demonstrate New Technique for Stealing AI Models" by Matt Shipman"How Cybercriminals Used AI To Mimic CEO's Voice To Steal £220,000" by Think Cloud"The rise of AI fake news is creating a 'misinformation superspreader'" by Pranshu Verma"A Comprehensive Guide to Access and Secrets Management: From Zero Trust to AI Integration — Innovations in Safeguarding Sensitive Information" by Boris Zaikin This is an excerpt from DZone's 2025 Trend Report, Generative AI: The Democratization of Intelligent SystemsRead the Free Report

By Boris Zaikin

CORE

Modern QA Practices to Enhance Software Reliability

Imagine a scenario: You are working at breakneck speed to roll out a new IT product or a business-critical update, but quality control workflows lack efficiency. They are mainly manual and performed late in the development cycle. This challenges your ability to cope with pressing deadlines, innovate, and remain competitive, as manual efforts are slow, workflows lack fast and continuous feedback loops, and critical and major issues arise just before the release. Transitioning to more advanced testing practices can turn the tide and positively impact operational productivity due to early defect detention, increased test coverage, and better software stability, allowing businesses to economize, ship high-quality IT solutions faster, and elevate end-user satisfaction. So, let’s have a closer look at these state-of-the-art QA approaches and their tangible gains. The What, When, and Why of Switching to Revamped Testing Practices Upgrading QA approaches occurs when companies overhaul the way testing activities are performed on their projects and switch to cutting-edge techniques, instruments, or frameworks, thus embracing more streamlined, robust, and cohesive quality control. Several obvious signs can signal that it’s time to shift to advanced testing methods, thus addressing the hurdles of a modern software engineering landscape: Overburdened project teams find it harder to cope with the constantly increasing number of repeated manual activities, such as regression testing. Frequent software issues in the production environment negatively affect the overall quality level and user experience. Rapid software releases make it difficult for QA engineers to handle the required amount of verification efforts and obtain feedback. Safeguarding software and confidential user information to prevent security incidents and ensure compliance with industry regulations is critical. Misunderstandings because of a lack of efficient communication strategies appear, which can provoke missed deadlines and worsen team morale. Significant company changes influencing all business workflows, including quality assurance, take place, meaning that modifications in structure, teams, and responsibilities can provoke a reassessment of how QA is handled. When embracing enhanced QA approaches, businesses can get numerous benefits, such as faster and smoother releases of failsafe software with fewer issues in the production environment. These IT products foster end-user satisfaction and minimize the risks of customer attrition. In addition, early identification of issues in the development process helps minimize them easier and prevents increased operational costs. What’s more, when all processes on the project operate smoothly and teams aren’t overburdened, they don’t face burnout or frustration and are less likely to leave the team. Top 5 Crucial Enablers for Company-Wide QA Modernization Let’s analyze what state-of-the-art approaches organizations can embrace to keep up with the modern development pace and facilitate innovation. 1. Shift-Left Testing Conventional methods of quality control typically occupy the last stage in the development process, just before the release. Although these methods serve to confirm the software meets established requirements, they detect glitches too late, significantly increasing time and financial efforts required for their elimination. In contrast, shift-left testing, which often goes hand in hand with test automation, is about moving QA as early as possible in the SDLC, ideally conducting it as soon as the first development activities take place. A highly beneficial practice for all project members, it allows finding and resolving issues when they are cost-effective to rectify, obtaining immediate response to the outcomes of any code adjustments, and ensuring more stable and predictable software releases. 2. Continuous Testing Continuous testing is another advanced QA practice in which QA engineers conduct ongoing verifications throughout the development timeline — from requirements to post-deployment phases. This method relies heavily on automated testing workflows seamlessly incorporated into CI/CD frameworks. It helps ensure that IT products always meet all functional and non-functional requirements and are ready to go live at a moment’s notice, thus reducing the probability of last-minute changes. In addition, as verifications are conducted uninterruptedly, team members get quick insights into the impact of all code alterations. 3. AI-Powered Automated QA By incorporating AI/ML mechanisms into automated workflows, QA teams can attain more intelligent, streamlined, and high-performing testing processes. For instance, they serve to identify scripts and parts that are most prone to glitches after code changes and prioritize test execution accordingly, craft life-like test data, and offer self-repair functions. These specifics amplify the testing scope, safeguard the process against human faults, reduce the complexity of maintaining test scripts, and accelerate QA velocity. 4. Live Environment Verification This modern practice requires QA teams to run verifications after software deployment, leveraging authentic scenarios and ongoing user activity. During such testing, QA engineers can leverage two similar production environments: the first one for active users and the second for verifying any updates. After a successful quality control process, all users are redirected to the environment with already verified software. Combined with shift-left testing, this approach brings various benefits for businesses, such as gauging software’s behavior under authentic circumstances, ongoing surveillance for glitches, and the possibility of easily and swiftly reversing the software rollout in case of problems. 5. Chaos Engineering QA engineers can leverage this advanced testing practice to boost system robustness and resilience to unexpected scenarios. Testing specialists deliberately introduce controlled disruptions into the software such as service shutdowns, database connection failures, memory spikes, to assess how the system handles the pressure and to detect vulnerabilities for subsequent fixing. This approach proactively eliminates system weaknesses, minimizes the chances of downtime after going live, and confirms that recovery mechanisms operate flawlessly. Leaving Conventional Testing Behind Although traditional QA approaches are effective for enhancing the quality of software solutions, there are cases where their capabilities fall short. In such instances, QA engineers must turn to more advanced methods, such as continuous and shift-left testing, AI-driven automated QA, post-production verification, and chaos engineering. These innovative techniques enable teams to maintain a regular release cadence, optimize operational expenses, and develop software that end users better receive.

By Pavel Novik

Using Terraform Moved Block to Refactor Resources

Terraform introduced the moved block in version 1.1.0. This block provides a straightforward way to refactor resources by explicitly mapping old resource addresses to new ones. It significantly reduces the risk of losing state or manually managing imports during renames or moves. In this article, we’ll explain what the moved block is and how to use it to streamline resource restructuring for smoother and safer Terraform updates. What Is the Terraform Moved Block? A Terraform moved block is used within a module’s moved section to declare the migration of a resource or data source from one address to another. It is helpful for refactoring Terraform code while ensuring that state information is preserved, preventing Terraform from destroying and recreating resources unnecessarily. The moved block also handles resource renaming or movement across modules, making state management seamless. The syntax of the Terraform moved block is as follows: Plain Text moved { from = "<old_address>" to = "<new_address>" } Where: from is the original address of the resource or module in the previous configurationto is the new address of the resource or module in the updated configuration When to Use the Terraform Moved Block? The moved block in Terraform is used when you need to refactor or rename resources in your configuration without destroying and recreating them. It can be useful for: Renaming resources: When you change the name of a resource within the configuration (e.g., aws_instance.old_name to aws_instance.new_name).Reorganizing modules: If a resource is moved between modules or changed from a root-level resource to a module-managed resource (or vice versa), you can use the moved block to keep Terraform aware of the transition.Refactoring module names: When you are renaming or reorganizing modules to align with new naming conventions or standards.Changing resource block types: If you change the type of a resource block (e.g., from google_compute_instance to google_compute_engine), the moved block maps the old resource to the new type. Note that this is possible only if the configurations are compatible and the resource’s state remains valid. Splitting or consolidating configurations: When you split a configuration into multiple files or modules or consolidate it into fewer files, use the moved block to reflect these changes without resource re-creation. How to Use the Terraform Moved Block Let’s consider some examples: Example 1: Renaming a Resource In this example, we will rename a Terraform resource from aws_instance.old_name to aws_instance.new_name. Plain Text # Old Configuration # This block is no longer present resource "aws_instance" "old_name" { ami = "ami-12345678" instance_type = "t2.micro" } # New Configuration resource "aws_instance" "new_name" { ami = "ami-12345678" instance_type = "t2.micro" } # Moved Block moved { from = "aws_instance.old_name" to = "aws_instance.new_name" } Here, the moved block tells Terraform that the resource aws_instance.old_name has been renamed to aws_instance.new_name. Note: After making these changes, make sure you run terraform plan to verify that Terraform recognizes the migration correctly and that no unnecessary changes will be applied. Example 2: Moving Resources Between Modules In the next example, a resource is moved from the root module to a nested module. We want the aws_s3_bucket.my_bucket resource to be moved from the root module to a module named storage: Plain Text # Old Configuration (Root Module) # This block is no longer present resource "aws_s3_bucket" "my_bucket" { bucket = "example-bucket" } # New Configuration (Nested Module) # modules/storage/main.tf resource "aws_s3_bucket" "my_bucket" { bucket = "example-bucket" } # Root Module Configuration module "storage" { source = "./modules/storage" } # Moved Block moved { from = "aws_s3_bucket.my_bucket" to = "module.storage.aws_s3_bucket.my_bucket" } The moved block updates the Terraform state to reflect the new location without destroying and recreating the resource. Example 3: Using Moved With for_each We will now move a resource to a new name using a for_each loop, which creates multiple resources dynamically. Plain Text # Old Configuration # This block is no longer present resource "aws_security_group" "old_group" { for_each = toset(["web", "db"]) name = "sg-${each.key}" } # New Configuration resource "aws_security_group" "new_group" { for_each = toset(["web", "db"]) name = "sg-${each.key}" } # Moved Block moved { from = "aws_security_group.old_group[\"web\"]" to = "aws_security_group.new_group[\"web\"]" } moved { from = "aws_security_group.old_group[\"db\"]" to = "aws_security_group.new_group[\"db\"]" } Each resource in the old_group is mapped to a corresponding resource in new_group using moved blocks. This ensures a smooth state migration for all instances of the resources without unnecessary recreation. Example 4: Moving a Resource With a Changed Identifier In this example, a resource’s identifier changes because the for_each key expression is modified. We want to rename the for_each keys from ["app", "db"] to ["frontend", "database"]. Without the moved block, Terraform would see these as new resources and destroy the old ones. Plain Text # Old Configuration resource "aws_security_group" "sg" { for_each = toset(["app", "db"]) name = "sg-${each.key}" } # New Configuration resource "aws_security_group" "sg" { for_each = toset(["frontend", "database"]) # Changed identifiers name = "sg-${each.key}" } # Moved Blocks moved { from = "aws_security_group.sg[\"app\"]" to = "aws_security_group.sg[\"frontend\"]" } moved { from = "aws_security_group.sg[\"db\"]" to = "aws_security_group.sg[\"database\"]" } Each moved block maps the old key (app or db) to the new key (frontend or database). When you run terraform plan and terraform apply, Terraform updates the state to match the new identifiers, avoiding unnecessary destruction and re-creation of the security groups. Example 5: Moving a Resource Between Providers If we want to migrate a resource from one provider to another (e.g., from aws to aws.other_region), the Terraform moved block ensures a seamless state transition. In our example, the S3 bucket resource is initially in the us-west-1 region using the default aws provider. We want to manage the same bucket in a different AWS region (e.g., us-east-1) using a different provider alias. Plain Text # Old Configuration provider "aws" { region = "us-west-1" } resource "aws_s3_bucket" "example_bucket" { bucket = "example-bucket" } # New Configuration provider "aws" { region = "us-west-1" } provider "aws" { alias = "other_region" region = "us-east-1" } resource "aws_s3_bucket" "example_bucket" { provider = aws.other_region bucket = "example-bucket" } # Moved Block moved { from = "aws_s3_bucket.example_bucket" to = "aws_s3_bucket.example_bucket" } The moved block ensures Terraform updates the state to associate the resource with the new provider configuration (aws.other_region). The bucket itself remains intact in AWS and is not recreated, avoiding downtime or data loss. Terraform Moved Block Limitations The moved block in Terraform is useful for handling resource renaming or moving across modules in a safe and structured manner, but it has some limitations: Manual specification. The moved block requires you to manually specify both the source and destination addresses, which can lead to human error if the information is not provided accurately.Resource renames only. The moved block is specifically designed for renaming or moving resources within the state. Splitting or restructuring state might require manual state modifications or the use of tools like terraform state mv.Static declaration. The moved block is static and does not support conditional logic. It cannot handle scenarios in which the movement depends on dynamic conditions.Limited to planned changes. The moved block is only applied during the terraform plan and apply commands. If you modify the state file manually, the moved block will not help reconcile those changes.Requires state compatibility. The source and destination must be part of the same state file. This is particularly important for users working with remote backends.Only supported in Terraform 1.1+. Older versions of Terraform do not recognize the moved block, making it incompatible with legacy configurations or environments. Key Points The moved block in Terraform is a practical tool for restructuring resources without disrupting infrastructure. It ensures seamless state updates, avoids downtime, and simplifies complex refactoring tasks, allowing you to reorganize code confidently and maintain operational stability — provided the mappings are accurate and proper planning is in place.

By Mariusz Michalowski

Avoid Cross-Shard Data Movement in Distributed Databases

Modern applications rely on distributed databases to handle massive amounts of data and scale seamlessly across multiple nodes. While sharding helps distribute the load, it also introduces a major challenge — cross-shard joins and data movement, which can significantly impact performance. When a query requires joining tables stored on different shards, the database must move data across nodes, leading to: High network latency due to data shufflingIncreased query execution time as distributed queries become expensiveHigher CPU usage as more computation is needed to merge data across nodes But what if we could eliminate unnecessary cross-shard joins? In this article, we'll explore four proven strategies to avoid data movement in distributed databases: Replicating reference tablesCollocating related data in the same shardUsing a mapping table for efficient joinsPrecomputed join tables (materialized views) By applying these strategies, you can optimize query execution, minimize network overhead, and scale your database efficiently. Understanding the Problem: Cross-Shard Joins and Data Movement How Data Movement Happens Imagine a typical e-commerce database where: The Orders table is sharded by CustomerID. SQL CREATE TABLE Orders ( order_id INT PRIMARY KEY AUTO_INCREMENT, -- Unique ID for the order customer_id INT NOT NULL, -- Foreign key to Customers (Sharding Key) product_id INT NOT NULL, -- Foreign key to Products quantity INT DEFAULT 1, -- Number of units ordered total_price DECIMAL(10,2), -- Total price of the order order_status ENUM('Pending', 'Shipped', 'Delivered', 'Cancelled') DEFAULT 'Pending', order_date TIMESTAMP DEFAULT CURRENT_TIMESTAMP, shard_key INT GENERATED ALWAYS AS (customer_id) VIRTUAL -- Helps with sharding logic ); Column Name Data Type Description order_id INT (Primary Key, Auto Increment) Unique identifier for the order. customer_id INT (Sharding Key) Foreign key to Customers table; used for sharding. product_id INT (Foreign Key) References Products.product_id. quantity INT (Default: 1) Number of units ordered. total_price DECIMAL(10,2) Total cost of the order. order_status ENUM('Pending', 'Shipped', 'Delivered', 'Cancelled') (Default: 'Pending') Current order status. order_date TIMESTAMP (Default: CURRENT_TIMESTAMP) Timestamp when the order was placed. shard_key INT (Virtual Column) Helps in sharding logic by using customer_id. Sharding Key: customer_id Ensures that all orders of a given customer remain on the same shard. Why Not order_id? Because customers place multiple orders, keeping all their orders on the same shard optimizes query performance for customer history retrieval. The Products table is sharded by ProductID. SQL CREATE TABLE Products ( product_id INT PRIMARY KEY AUTO_INCREMENT, -- Unique product ID (Sharding Key) product_name VARCHAR(255) NOT NULL, -- Name of the product category VARCHAR(100), -- Product category (e.g., Electronics, Clothing) price DECIMAL(10,2) NOT NULL, -- Product price stock_quantity INT DEFAULT 0, -- Available stock count last_updated TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, shard_key INT GENERATED ALWAYS AS (product_id) VIRTUAL -- Helps with sharding logic ); Column Name Data Type Description product_id INT (Primary Key, Auto Increment) Unique identifier for the product (Sharding Key). product_name VARCHAR(255) Name of the product. category VARCHAR(100) Product category (e.g., Electronics, Clothing). price DECIMAL(10,2) Price of the product. stock_quantity INT (Default: 0) Available stock count. last_updated TIMESTAMP (Default: CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP) Timestamp for last update. shard_key INT (Virtual Column) Helps in sharding logic by using product_id. Sharding Key: product_id Ensures that product details are distributed evenly across shards. Why Not category? Because product_id ensures uniform distribution, whereas some categories may be more popular and could cause unbalanced shards. Now, let's say we want to retrieve the product names for a specific customer’s orders: SQL SELECT o.order_id, o.customer_id, p.product_name FROM Orders o JOIN Products p ON o.product_id = p.product_id WHERE o.customer_id = 123; Since Orders and Products are sharded differently, fetching the product_name requires data movement between shards. This can cause significant performance degradation, especially as the system scales. Strategy 1: Replicating Reference Tables Concept Instead of storing reference tables on separate shards, we replicate them across all nodes. This allows queries to perform joins locally, eliminating inter-shard communication. How to Implement In a database like PostgreSQL Citus, we can create a replicated reference table: SQL CREATE TABLE Products ( product_id INT PRIMARY KEY, product_name TEXT ) DISTRIBUTED REPLICATED; Benefits Eliminates cross-shard joins for lookups.Fast query execution, as the reference data is available on all shards. When NOT to Use If the reference table is too large (e.g., millions of records).If the reference data changes frequently, causing replication overhead. Strategy 2: Collocating Related Data in the Same Shard Concept Instead of sharding Orders by CustomerID and Products by ProductID, we shard both tables by the same key (CustomerID). How It Works SQL SELECT o.order_id, o.customer_id, p.product_name FROM Orders o JOIN Products p ON o.product_id = p.product_id WHERE o.customer_id = 123; Now, because both tables are in the same shard, the join happens locally, avoiding cross-shard data movement. Benefits Optimized for queries that frequently join tables based on a shared key.Works well for applications where customer-specific data is frequently accessed. When NOT to Use If tables grow at different rates, leading to uneven shard distribution.If queries involve cross-customer searches, making sharding by CustomerID inefficient. Strategy 3: Using a Mapping Table for Efficient Joins Concept Instead of performing cross-shard joins, we use a mapping table that stores which shard contains the required data. How It Works We create a customer-product mapping table: SQL CREATE TABLE CustomerProductMap ( customer_id INT, product_id INT, shard_id INT, PRIMARY KEY (customer_id, product_id) ); Before querying, we first retrieve the shard_id: SQL SELECT shard_id FROM CustomerProductMap WHERE customer_id = 123; Then, we query only the relevant shard: SQL SELECT o.order_id, o.customer_id, p.product_name FROM Orders o JOIN Products p ON o.product_id = p.product_id WHERE o.customer_id = 123 AND o.shard_id = <retrieved_shard>; Benefits Reduces the number of shards queried, improving performance.Works well for large datasets that cannot be replicated. When NOT to Use If updates to CustomerProductMap are frequent, causing maintenance overhead. Strategy 4: Precomputed Join Tables (Materialized Views) Concept Instead of executing joins at query time, we precompute and store frequently used join results in a materialized view or a separate table. How It Works We create a denormalized table that precomputes Orders and Products: SQL CREATE TABLE Precomputed_Order_Products ( order_id INT PRIMARY KEY, customer_id INT, product_id INT, product_name TEXT, order_date TIMESTAMP ) SHARDED BY (customer_id); A background ETL process precomputes the joins and inserts the data: SQL INSERT INTO Precomputed_Order_Products (order_id, customer_id, product_id, product_name, order_date) SELECT o.order_id, o.customer_id, p.product_id, p.product_name, o.order_date FROM Orders o JOIN Products p ON o.product_id = p.product_id; Querying without joins: SQL SELECT order_id, customer_id, product_name, order_date FROM Precomputed_Order_Products WHERE customer_id = 123; Benefits No joins required at query time, significantly improving performance.Ideal for read-heavy workloads where query speed is critical. When NOT to Use If the dataset changes frequently, requiring frequent recomputation.If storage costs for redundant data are too high. Choosing the Right Strategy: Trade-offs and Considerations ApproachProsConsReplicating Reference TablesFast joins, eliminates data movementOnly works for small tablesCollocating Data in the Same ShardEfficient joins, avoids movementRequires careful shard key selectionMapping Table ApproachEfficient query routingExtra storage & maintenance overheadPrecomputed Join TablesBest for read-heavy workloadsRequires background ETL jobs General Guidelines Use replication – For small reference tablesUse colocation – When queries naturally group by a specific keyUse mapping tables – When reference tables are too large to replicateUse precomputed joins – For high-performance analytical queries Conclusion Cross-shard data movement is one of the biggest performance challenges in distributed databases. We can significantly optimize query performance and reduce unnecessary network overhead by applying reference table replication, colocation, mapping tables, and precomputed joins. Choosing the right strategy depends on query patterns, data distribution, and storage constraints. With the right approach, you can build highly scalable, performant distributed systems without being hindered by cross-shard joins.

By Baskar Sikkayan

Jakarta NoSQL 1.0: A Way To Bring Java and NoSQL Together

NoSQL is no longer the exception — it's a key part of modern data architectures. With the release of Jakarta NoSQL 1.0, Java developers finally gain a standardized, extensible way to interact with NoSQL databases across document, key-value, column, and graph data models — all while staying aligned with Jakarta EE principles. This article introduces the Jakarta NoSQL 1.0 specification, which boosts productivity, simplifies mapping, and provides fluent, type-safe access to NoSQL data. The Goal: Simplifying Java-NoSQL Integration Jakarta NoSQL is the first specification in the Jakarta EE ecosystem designed to integrate Java applications with NoSQL technologies in a portable and standard way. The specification was built with a few core goals in mind: Increase developer productivity when working with NoSQL databasesEnable annotation-driven, rich object mapping similar to Jakarta PersistenceProvide a Java-based query and fluent API for seamless database operations Annotation-Based Mapping One of Jakarta NoSQL's core strengths is its annotation model, which borrows familiar concepts from Jakarta Persistence (JPA). This means that developers already familiar with JPA can get started quickly. The key annotations include: AnnotationDescription@EntityMarks a class as persistable@IdDefines the primary key@ColumnMaps fields or properties@Embeddable, @MappedSuperclassEnable composition and inheritance@ConvertDefines how to persist custom types Here’s a basic example of a Car entity: Java @Entity public class Car { @Id private Long id; @Column private String name; @Column private CarType type; // Getters and setters... } Note: Jakarta NoSQL intentionally does not define relationship mappings such as @OneToMany or @ManyToOne Due to the nature of NoSQL models, implementations can offer such extensions when the database supports them. The Template Interface: A Unified Data Bridge Jakarta NoSQL introduces the Template interface to connect Java entities with NoSQL databases. Through this interface, you can perform standard CRUD operations and fluent queries with a consistent API. Basic operations look like this: Java @Inject Template template; Car car = Car.id(1L) .name("Ferrari") .type(CarType.SPORT); template.insert(car); Optional<Car> result = template.find(Car.class, 1L); template.delete(Car.class, 1L); Querying With Fluent API In addition to essential persistence, the Template Interface offers a fluent querying API, making NoSQL access expressive and type-safe: Java List<Car> cars = template.select(Car.class) .where("type").eq(CarType.SUV) .orderBy("name").asc() .result(); template.delete(Car.class) .where("type").eq(CarType.COUPE) .execute(); This abstraction makes it easy to express filters, sort orders, and actions without writing vendor-specific query syntax. Extensibility and Implementation Jakarta NoSQL is extensible by design. While the spec defines the standard behavior, NoSQL providers can add their specific capabilities through extensions, including custom queries, transactions, or even relationship support. The reference implementation, Eclipse JNoSQL, offers: Full support for Jakarta NoSQL annotations and templatesIntegration with CDI and CDI LiteAnnotation processors to reduce reflection usage in native or cloud environments Tooling support is also available via IntelliJ IDEA plugins to recognize entities and fields. Supporting Jakarta NoSQL Trade-Offs of Using Jakarta NoSQL While Jakarta NoSQL brings a welcome level of standardization and consistency to the Java-NoSQL landscape, it’s essential to understand the trade-offs involved when adopting this specification: Pros Familiar programming model. Developers already using Jakarta Persistence (JPA) will find the annotation model intuitive and easy to adopt.Portability. The abstraction allows you to switch between NoSQL databases with minimal code changes.Extensibility. Vendors can provide extensions for advanced or database-specific behavior, enabling the API to grow with your needs.Integration with Jakarta EE. Fits naturally into Jakarta EE projects using CDI, making it a first-class citizen in the enterprise Java world.Improved developer productivity. Rich object mapping and fluent query APIs simplify the learning curve and reduce boilerplate. Cons and Limitations Abstraction overhead. The API's generic nature may hide database-specific features, limiting access to native capabilities unless you use vendor extensions.No relationship support by default. Unlike JPA, there’s no built-in support for entity relationships (like @ManyToOne), as many NoSQL databases don’t support joins. This can lead to more manual modeling and denormalized data.Learning curve for NoSQL modeling. Developers coming from relational backgrounds might need to adapt to different modeling approaches, especially in document and graph databases.Performance tuning limitations. Optimizing queries might require dropping down to native APIs when you need fine-grained control, which can reduce portability. In summary, Jakarta NoSQL offers an elegant, standardized path for integrating Java with NoSQL, but like any abstraction, it requires thoughtful consideration when choosing between portability and database-specific optimization. Conclusion Jakarta NoSQL 1.0 fills a long-standing gap in the Java ecosystem, offering a standard, vendor-neutral way to work with NoSQL technologies using Jakarta EE patterns. Whether you're building cloud-native microservices or evolving legacy systems toward modern data models, Jakarta NoSQL provides a foundation on which to grow. If you're a Java developer already using NoSQL, Jakarta NoSQL makes your work more portable and productive. And if you're considering transitioning from relational to NoSQL, this spec provides a familiar and consistent path forward. You can explore the full specifications here. Further Reading Eclipse JNoSQL ProjectJakarta EE Specifications

By Otavio Santana

CORE

Building AI-Driven Intelligent Applications: A Hands-On Development Guide for Integrating GenAI Into Your Applications

Editor's Note: The following is an article written for and published in DZone's 2025 Trend Report, Generative AI: The Democratization of Intelligent Systems. In today's world of software development, we are seeing a rapid shift in how we design and build applications, mainly driven by the adoption of generative AI (GenAI) across industries. These intelligent applications can understand and respond to users' questions in a more dynamic way, which help enhance customer experiences, automate workflows, and drive innovation. Generative AI — mainly powered by large language models (LLMs) like OpenAI GPT, Meta Llama, and Anthropic Claude — has changed the game by making it easy to understand natural language text and respond with new content such as text, images, audio, and even code. There are different LLMs available from various organizations that can be easily plugged into existing applications based on project needs. This article explains how to build a GenAI-powered chatbot and provides a step-by-step guide to build an intelligent application. Why There Is a Need for GenAI Integration Within the field of artificial intelligence (AI), GenAI is a subset of deep learning methods that can work with both structured and unstructured data. Figure 1 presents the different layers of artificial intelligence. Figure 1. Artificial intelligence landscape It is notably generative AI that is rapidly transforming the business world by fundamentally reshaping how businesses automate tasks, enhance customer interactions, optimize operations, and drive innovation. Companies are embracing AI solutions to have a competitive edge, streamline workflows, and unlock new business opportunities. As AI becomes more woven into society, its economic impact will be significant, and organizations are already starting to understand its full potential. According to the MIT Sloan Management Review, 87% of organizations believe AI will give them a competitive edge. McKinsey's 2024 survey also revealed that 65% of respondents reported their organizations are regularly using GenAI — nearly double the percentage from the previous survey 10 months prior. Figure 2 lays out a comparison of the shift in organizations that are building intelligent applications. Figure 2. Comparison between traditional and intelligent apps Advantages of GenAI Integration Integrating GenAI into applications brings several advantages and revolutionizes how businesses operate: AI models can analyze vast amounts of structured and unstructured data to extract valuable insights, enabling data-driven decision making.By automating intensive, repetitive tasks and therefore improving efficiency, businesses can reduce operational costs.GenAI enables real-time personalization and makes interactions more engaging. For example, GenAI-powered chatbots can recommend products based on the user's browsing history.GenAI can be a powerful tool for brainstorming and content creation, which empowers businesses to develop innovative products, services, and solutions.GenAI-powered chatbots can provide instant customer support by answering questions and resolving issues. This can significantly improve customer satisfaction, reduce wait times, and free up human agents' time to handle more complex inquiries. Precautions for Integrating GenAI While integrating GenAI to build intelligent applications, solutions, or services, organizations need to be aware of challenges and take necessary precautions: AI models can generate misleading or factually incorrect responses, often called hallucinations. This leads to poor decision making and legal or ethical implications, which can damage an organization's reputation and trustworthiness.Because AI models are trained on historical data, they can inherit biases, leading to unfair or discriminatory outcomes. It's crucial to embrace responsible AI practices to evaluate and mitigate any potential biases.Integrating GenAI into existing systems and workflows can become complicated and requires significant technical expertise.As governments around the world are introducing new AI regulations, companies must stay up to date and implement AI governance frameworks to meet legal and ethical standards.Growing threats of deep fakes, misinformation, and AI-powered cyberattacks can undermine public trust and brand reputation. To counter these risks, businesses should invest in AI moderation tools and adopt proactive strategies to detect and mitigate harmful or misleading content before it reaches users. Through strong governance, ethical frameworks, and continuous monitoring, organizations can unlock the potential of AI while protecting their operations, trust, and customer data. Tech Stack Options There are different options available if you are considering integrating GenAI to build intelligent applications. The following are some, but not all, popular tool options: Open-source tools: Microsoft Semantic Kernel is an AI orchestration framework for integrating LLMs with applications using programming languages like C#, Python, and Java.LangChain is a powerful framework for chaining AI prompts and memory.Hugging Face is a library for using and fine-tuning open-source AI models.LlamaIndex is a framework that provides a central interface to connect your LLM with external data sources.Enterprise tools: OpenAI API provides access to OpenAI's powerful models, including GPT-3, GPT-4o, and others.Azure AI Foundry is Microsoft's cloud-based AI platform that provides a suite of services for building, training, and deploying AI models.Amazon SageMaker is a fully managed service for building, training, and deploying machine learning models.Google Cloud Vertex AI is a platform for building and deploying machine learning models on Google Cloud. Deciding whether to choose an open-source or enterprise platform to build intelligent AI applications depends on your project requirements, team capabilities, budget, and technical expertise. In some situations, a combination of tools from both open-source and enterprise ecosystems can be the most effective approach. How to Build a GenAI-Powered Customer Chatbot Regardless of the tech stack you select, you will have to follow the same steps to integrate AI. In this section, I will focus on building a chatbot that takes any PDF file, extracts data, and chats with the user by answering questions. It can be hosted on a web application as an e-commerce chatbot to answer user inquiries, but due to the size limitations, I will create it as a console application. Step 1. Setup and Prerequisites Before we begin, we'll need to: Create a personal access token (PAT), which helps to authenticate 1,200+ models that are available in GitHub Marketplace. Instead of creating developer API keys for every model, you can create one PAT token and use it to call any model.Install OpenAI SDK using pip install openai (requires: Python ≥ 3.8).Have an IDE platform for writing Python code. Not mandatory, however, I will use Visual Studio Code (VSCode) to write code in this example.Download and install Python.Download this VSCode extension, which offers rich support for the Python language; it also offers debugging, refactoring, code navigation, and more.Install PDFPlumber using pip install pdfplumber, a Python library that allows you to extract text, tables, and metadata from PDF files. Step 2. Create Python Project and Set Up a Virtual Environment Using VSCode or your favorite IDE, create a folder named PythonChatApp. Ensure that Python is installed on your system. Navigate to the project root directory and run the below commands to create a virtual environment. Creating a virtual environment is optional but recommended to isolate project environments. Shell Pip install Virtualenv cd c:\Users\nagavo\source\repos\PythonChatApp virtualenv myenv Step 3. Create Github_Utils.py File to Interact With OpenAI Models The Github_Utils.py file imports the OpenAI library and sets up required global variables. An OpenAI client is instantiated with a base URL and API PAT token. This client becomes the interface to send requests to the API. One advantage with GitHub Models is that we can easily switch to the GPT o3-mini model by simply updating the model name variable without changing existing code. This file contains two functions, summarize_text and ask_question_about-text, both of which are responsible for summarizing the text from the selected PDF and, later on, to ask questions related to the content. The file contents are shown below: Python import os from openai import OpenAI # Load GitHub API PAT Token and endpoint github_PAT_key = "<paste your github PatToken>" github_api_url = "https://models.inference.ai.azure.com" model_name = "gpt-4o" # Initialize OpenAI client client = OpenAI( base_url=github_api_url, api_key=github_PAT_key, ) def summarize_text(text): """Summarizes the text using GitHub's models API.""" try: response = client.chat.completions.create( messages=[ { "role": "system", "content": "You are a helpful assistant.", }, { "role": "user", "content": f"Summarize the following text: {text}", } ], temperature=0.7, top_p=1.0, max_tokens=300, model=model_name ) return response.choices[0].message.content except Exception as e: return f"Error with GitHub API: {e}" def ask_question_about_text(text, question): """Asks a question about the text using GitHub's models API.""" try: response = client.chat.completions.create( messages=[ { "role": "system", "content": "You are a helpful assistant.", }, { "role": "user", "content": f"Based on the following text: {text}\n\n Answer this question: {question}", } ], temperature=0.7, top_p=1.0, max_tokens=300, model=model_name ) return response.choices[0].message.content except Exception as e: return f"Error with GitHub API: {e}" Step 4. Create Pdf_Utils.py File to Extract Data From PDF The Pdf_Utils.py file contains utility functions for working with PDF files, specifically for extracting text from them. The main function, extract_text_from_pdf, takes the path to a PDF file as an argument and returns the extracted text as a string by utilizing the PDFPlumber library. The file contents are shown on the following page. Python try: import pdfplumber except ImportError: print("Error: pdfplumber module not found. Please install it using 'pip install pdfplumber'.") def extract_text_from_pdf(pdf_path): """Extracts text from a PDF file.""" text = "" try: with pdfplumber.open(pdf_path) as pdf: for page in pdf.pages: page_text = page.extract_text() if page_text: text += page_text + "\n" except Exception as e: print(f"Error reading PDF: {e}") return text Step 5. Create Main.Py, an Entry Point for the Application The Main.Py file acts as the main entry point for the application. It is responsible for receiving user input to process the PDF file and interacting with the AI service. It imports Pdf_utils and Github_utils to interact with methods in both files. The file contents of Main.py are shown below: Python import os from Pdf_utils import extract_text_from_pdf from Github_utils import summarize_text, ask_question_about_text def main(): print("=== PDF Chatbot ===") # Ask user to specify PDF file path pdf_path = input("Please enter the path to the PDF file: ").strip() if not os.path.isfile(pdf_path): print(f"Error: The file '{pdf_path}' does not exist.") return # Extract text from PDF print("\nExtracting text from PDF...") text = extract_text_from_pdf(pdf_path) if not text.strip(): print("No text found in the PDF.") return # Summarize the text print("\nSummary of the PDF:") try: summary = summarize_text(text) print(summary) except Exception as e: print(f"Error with GitHub API: {e}") return # Ask questions about the text while True: print("\nYou can now ask questions about the document!") question = input("Enter your question (or type 'exit' to quit): ").strip() if question.lower() == "exit": print("Exiting the chatbot. Goodbye!") break try: answer = ask_question_about_text(text, question) print("\nAnswer:") print(answer) except Exception as e: print(f"Error with GitHub API: {e}") break if __name__ == "__main__": main() Step 6. Run the Application Open terminal and navigate to the root directory. To run the application, enter the following command: C:\Users\nagavo\source\repos\PythonChatApp\PDF_Chatbot> python main.py Enter the path to the PDF file in the response. In this example, I uploaded a resume to help me summarize the candidate's profile and her experience. This file is sent to the GPT-4o model to summarize the file contents as shown in Figure 3. Figure 3. Summary of the PDF document in the GPT response Let's ask some questions about the document, for example, "How many years of experience does the candidate have with date ranges?" Figure 4 shows the detailed response from the chatbot application. Figure 4. Response from asking specific question about the document We can host this app on an e-commerce website to upload product and order information. This allows customers to interact by asking specific questions about products or their orders, thus avoiding customer agents manually answering these questions. There are multiple ways we can leverage GenAI across industries; this is just one example. Conclusion Integrating GenAI into applications is no longer a luxury but a necessity for businesses to stay competitive. The adoption of GenAI offers numerous advantages, including increased productivity, improved decision making, and cost savings by avoiding repetitive work. It is also crucial to be aware of the challenges and risks associated with GenAI, such as hallucination, bias, and regulatory compliance, as it is easy to misuse AI-generated content. It is essential to adopt responsible AI practices and invent robust governance frameworks to ensure ethical and fair use of AI technologies and by doing so, organizations can unlock the full potential of GenAI while protecting their reputation and trust from their customers. References: "AI regulations around the world: Trends, takeaways & what to watch heading into 2025" by Diligent Blog"Superagency in the workplace: Empowering people to unlock AI's full potential" by Hannah Mayer, et al."Expanding AI's Impact With Organizational Learning" by Sam Ransbotham, et al., MITSloan"Embracing Responsible AI: Principles and Practices" by Naga Santhosh Reddy Vootukuri This is an excerpt from DZone's 2025 Trend Report, Generative AI: The Democratization of Intelligent SystemsRead the Free Report

By Naga Santhosh Reddy Vootukuri

CORE

Java Reborn: Conquering Cloud-Native and Next-Gen Concurrency

Although Java has been around for long time, the future looks exciting. Historically being the go to language for building enterprise applications, the language has grown to match the speed of development of the cloud development technology. In this article, we take a closer look as to how Java keeps on evolving with focus on cloud native designs, improving concurrency, microservices architecture, and new initiatives that promise to change the way Java is seen and used. 1. Java's Ongoing Transformation Swift Release Cycles Incorporating Modern Features Over the years, the release frequency of Java has significantly improved, now with releases almost every six months. This keeps the Java community excited and consistently gives developers new capabilities and performance boosts. This is the strongest indicator that the language wants to stay modern and relevant in the ever-changing world of technology. For example, switch expression, introduced in Java 14, simplifies the way the switch statement is written. Java public String getDayType(int day) { return switch (day) { case 1, 7 -> "Weekend"; case 2, 3, 4, 5, 6 -> "Weekday"; default -> "Invalid day"; }; } A Mature But Adaptive Ecosystem Although used for building applications using monolith architecture, Java has come a long way. Newer libraries and frameworks are created to support cloud-based architectures, making Java a reliable option for creating microservices and distributed systems. A simple Spring Boot microservice can be written in as few lines as below: Java import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @SpringBootApplication @RestController public class MicroserviceDemo { public static void main(String[] args) { SpringApplication.run(MicroserviceDemo.class, args); } @GetMapping("/hello") public String sayHello() { return "Hello from a Java microservice!"; } } 2. Full Steam Ahead on Cloud-Native Microservices Done the Right Way Spring Boot, Quarkus, and Micronaut are just some of the frameworks redefining how Java does microservices. These frameworks prioritize swift startup, efficient resource use, and effortless scaling on Kubernetes, which perfectly aligns with container-based workflows. Error messages can be built into the microservices, allowing easier and customized error handling. Java import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.PathVariable; import org.springframework.web.bind.annotation.RestController; @RestController public class UserController { @GetMapping("/user/{id}") public String getUser(@PathVariable int id) { if (id <= 0) { throw new IllegalArgumentException("Invalid user ID"); } return "User " + id + " details"; } } Openly Embracing Serverless With integrations available for AWS Lambda, Azure Functions, and Google Cloud Functions, Java has become a serious contender in the serverless applications space. Quarkus, in particular, reduces the startup time and removes the stigma of Java being heavy for function-as-a-service. The following snippet shows the capability of serverless applications using Quarkus and AWS Lambda. Java import io.quarkus.amazon.lambda.runtime.QuarkusLambda; import com.amazonaws.services.lambda.runtime.Context; import com.amazonaws.services.lambda.runtime.RequestHandler; public class HelloLambda implements RequestHandler<String, String>, QuarkusLambda { @Override public String handleRequest(String input, Context context) { return "Hello, " + (input != null ? input : "World") + "!"; } } Containers and Seamless DevOps Due to turned down Apline builds and optimied JDKs, the image size for Java has significantly reduced. Continuing the trend of becoming a modern language, Java pipelines mesh smoothly with DevOps practices like integration, automated checks and container registry, which keeps deployments on track. 3. Project Loom: Rethinking Concurrency Virtual Threads: A New Approach Managing a lot of tasks in Java has always been challenging, given the need to manage multiple threads. However, this is about to change, with Project Loom introducing virtual threads, which offer a lightweight solution to tackle complex high-volume tasks without affecting performance. An example of how virtual threads can be used to handle multiple tasks effectively is shown below. Java import java.util.concurrent.Executors; public class VirtualThreadDemo { public static void main(String[] args) throws InterruptedException { try (var executor = Executors.newVirtualThreadPerTaskExecutor()) { for (int i = 0; i < 10_000; i++) { executor.submit(() -> { System.out.println("Task running in virtual thread: " + Thread.currentThread()); try { Thread.sleep(100); } catch (Exception e) {} }); } } System.out.println("All tasks submitted!"); } } Structured Concurrency: Clarity in Complexity Project Loom also introduced 'structured concurrency', which allows the organization and management of multiple tasks at the same time, which certainly will make code neat and easy to work with. While structured concurrency is still evolving, we can stimulate it using StructuredTaskForce, as shown below. Java import jdk.incubator.concurrent.StructuredTaskScope; public class StructuredConcurrencyDemo { public static void main(String[] args) throws Exception { try (var scope = new StructuredTaskScope.ShutdownOnFailure()) { var task1 = scope.fork(() -> { Thread.sleep(100); return "Result from Task 1"; }); var task2 = scope.fork(() -> { Thread.sleep(150); return "Result from Task 2"; }); scope.join(); // Wait for all tasks to complete scope.throwIfFailed(e -> new RuntimeException(e)); System.out.println(task1.get() + " | " + task2.get()); } } } 4. Pushing Performance Boundaries Valhalla: Leaner Memory Usage Project Valhalla introduces value types, which promise more effective handling of memory. This will come in handy for data-intensive process or performing intense cryptographic operations. Panama: Smoother Native Integration Project Panama aims to make it easier to use native code. For tasks that rely on GPU acceleration or specialized libraries, this will certainly boost application performance and make Java one of the contenders in building such applications. The following is an example of calling printf from C language in Java code using Panama's API. Java import java.lang.foreign.*; import java.lang.invoke.MethodHandle; public class PanamaDemo { public static void main(String[] args) { try (Arena arena = Arena.ofConfined()) { SymbolLookup stdlib = SymbolLookup.libraryLookup("c", arena); MethodHandle printf = Linker.nativeLinker().downcallHandle( stdlib.find("printf").orElseThrow(), FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.ADDRESS) ); MemorySegment format = arena.allocateUtf8String("Hello from Java via Panama! %d\n"); printf.invoke(format, 2025); } catch (Throwable t) { t.printStackTrace(); } } } 5. Thriving in the Microservices Era Smaller Footprints, Faster Launch Using tools such as GraalVM allows us to compile Java into native images, which will drastically cut down on startup times and the need for memory. This is vital for microservices architecture since the service needs to be available at very short notice. Observability Build In In a world as complex as microservices, visibility into what is happening under the hood is always handy. With the availability of OpenTelemetry and other similar tools, Java now allows developers to collect metrics, traces, and logs to keep the system healthy and troubleshoot issues in real time. The following snippet shows an example of adding OpenTelemetry tracking into a basic Spring Boot application, Java import io.opentelemetry.api.trace.Span; import io.opentelemetry.api.trace.Tracer; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @RestController public class TraceController { @Autowired private Tracer tracer; @GetMapping("/trace") public String tracedEndpoint() { Span span = tracer.spanBuilder("traced-endpoint").startSpan(); try (var scope = span.makeCurrent()) { span.addEvent("Processing request"); return "Request traced!"; } finally { span.end(); } } } Resilient by Design Java libraries available today come with fallback mechanisms like circuit breakers and retry logic. This allows for the graceful handling of services if there are performance issues without affecting the application as a whole. The following snippet shows the usage of Resilience4j in a Spring Boot application. Java import io.github.resilience4j.circuitbreaker.annotation.CircuitBreaker; import org.springframework.web.bind.annotation.GetMapping; import org.springframework.web.bind.annotation.RestController; @RestController public class ResilienceController { @GetMapping("/resilient") @CircuitBreaker(name = "myService", fallbackMethod = "fallback") public String callService() { // Simulate a failing service if (Math.random() > 0.5) { throw new RuntimeException("Service failed!"); } return "Service responded!"; } public String fallback(Throwable t) { return "Fallback response due to: " + t.getMessage(); } } 6. Tools and the Power of Community Updated Build Systems Maven and Gradle keep up with Java’s changes, allowing developers to use new language features quickly without having to redo their whole project setup. Smarter IDEs IDEs such as IntelliJ IDEA, Eclipse, or VS Code are providing advanced editing tools, error detection, and debugging options that greatly improve development speed A Global Network of Developers Java’s open-source community is still lively, providing quick updates, clear library guides, and resources that are easy for beginners to use. It is a big group that keeps pushing Java ahead. 7. Looking to the Horizon Java is focusing more on cloud systems, so it will work on starting faster and using fewer resources. At the same time, better multitasking from Loom, plus speed improvements from Valhalla and Panama, help Java handle everything from big business apps to new AI systems. Basically, Java shows it’s not just an old tool — it’s a leader. It keeps up with new needs, has a plan full of fresh ideas, and a community that never stops. Java is set to stay a key part of modern tech, from huge cloud projects to the newest tech breakthroughs. It’s steady, quick, and ready for the future. Summary With strong support from the community, Java continues to evolve, proving it is compatible with the cloud-driven world. Though historically used for enterprise application, with frequent release cycles and delivery of new developer-friendly features more frequently, Java is fast becoming the language of choice for a variety of use cases. The ecosystem where it is preferred is fast changing from monolity to cloud native architectures, with frameworks such as Spring Boot, Quarkus, and Micronaut leading the way. These frameworks also allow the microservices to scale efficiently with Kubernetes. Another feather in the cap is excelling in the serverless architecture, with tools like Quarkus reducing startup time for platforms like AWS Lambdas. Simplifying the management of multiple threads, Project Loom introduces virtual threads and structured concurrency, while Project Valhalla and Panama promise better memory management usage and native code integration. In the microservices era, Java offers native image compilation via GraalVM for faster launches, OpenTelemetry for observability, and resilience features like circuit breakers. With support from updated tools like Maven and Gradle, smarter IDEs like IntelliJ and Eclipse, and an ever-enthusiastic community, Java is well positioned to participate in technological advancements from cloud-native applications to AI-driven systems, remaining a reliable and innovative force in modern software development.

By Alok Tibrewala