Data Engineering Resources

DZone's Featured Data Engineering Resources

Running a Mobile App API Locally With Docker and Postman

By Swapnil Patil

In the development of mobile applications, a well-defined API is crucial for enabling seamless communication between the mobile front end and the backend services. Running this API locally can significantly enhance the development workflow, allowing developers to test and debug their applications without deploying them to a remote server. In this article, we will explore how to run a mobile app API locally using Docker and how to test it effectively with Postman. Why Use Docker? Docker provides a lightweight environment for running applications in containers, ensuring consistency across development, testing, and production environments. Using Docker, developers can isolate dependencies, manage versions, and streamline the deployment process. Setting Up the Project Before we begin, ensure that you have Docker and Postman installed on your machine. 1. Create a Simple API with Node.js and Express Let’s create a simple RESTful API using Node.js and Express. We will implement endpoints for a mobile app that manages a list of tasks. Step 1: Project Structure Create a new directory for your project: Shell mkdir mobile-app-api cd mobile-app-api mkdir src Inside the src directory, create the following files: server.jspackage.json Step 2: Define the API package.json: JSON { "name": "mobile-app-api", "version": "1.0.0", "description": "A simple API for managing tasks", "main": "server.js", "scripts": { "start": "node server.js" }, "dependencies": { "express": "^4.17.3" } } server.js: JavaScript const express = require('express'); const app = express(); const port = 3000; app.use(express.json()); let tasks = [ { id: 1, title: 'Task One', completed: false }, { id: 2, title: 'Task Two', completed: false }, ]; // Get all tasks app.get('/api/tasks', (req, res) => { res.json(tasks); }); // Create a new task app.post('/api/tasks', (req, res) => { const newTask = { id: tasks.length + 1, title: req.body.title, completed: false, }; tasks.push(newTask); res.status(201).json(newTask); }); // Start the server app.listen(port, () => { console.log(`API running at http://localhost:${port}`); }); Step 3: Create a Dockerfile In the root directory, create a file named Dockerfile: Dockerfile # Use the official Node.js image FROM node:18 # Set the working directory WORKDIR /usr/src/app # Copy package.json and install dependencies COPY src/package*.json ./ RUN npm install # Copy the rest of the application files COPY src/ . # Expose the API port EXPOSE 3000 # Start the application CMD ["npm", "start"] Step 4: Create a Docker Compose File To simplify running the API, create a docker-compose.yml file in the root directory: YAML version: '3.8' services: api: build: . ports: - "3000:3000" 2. Building and Running the API With Docker To build and run your API, execute the following commands in your terminal: Shell # Build the Docker image docker-compose build # Run the Docker container docker-compose up You should see output indicating that the API is running: Plain Text API running at http://localhost:3000 3. Testing the API With Postman Now that your API is running locally, you can test it using Postman. Step 1: Open Postman Launch Postman and create a new request. Step 2: Test the GET Endpoint Set the request type to "GET."Enter the URL: http://localhost:3000/api/tasks.Click "Send." You should see the list of tasks returned in JSON format. Step 3: Test the POST Endpoint Set the request type to "POST."Enter the URL: http://localhost:3000/api/tasks.In the Body tab, select "raw" and set the format to JSON.Enter the following JSON to create a new task: JSON { "title": "Task Three" } Click "Send." You should see the newly created task in the response. 4. Visualizing the API Architecture Here’s a simple diagram representing the architecture of our API: Plain Text ┌─────────────────────┐ │ Mobile App │ └──────────▲──────────┘ │ ┌──────────┴──────────┐ │ Postman Client │ └──────────▲──────────┘ │ ┌─────────┴───────────┐ │ Docker Container │ │ (Node.js + Express) │ └─────────▲───────────┘ │ ┌────────┴────────┐ │ API │ └─────────────────┘ Conclusion Running a mobile app API locally using Docker allows developers to create a consistent and isolated development environment. Using for containerization and Postman for testing, you can efficiently build, run, and debug your API, leading to a smoother development experience. This setup accelerates development and ensures that the application behaves consistently across various environments. Next Steps Explore Docker Networking to connect multiple services.Implement a database (e.g., MongoDB or PostgreSQL) for persistent storage.Integrate API documentation tools like Swagger for better API management.Consider deploying the API to a cloud service once it’s production-ready. You should now be able to streamline your mobile application development workflow, ensuring your API is robust and reliable. More

How to Build a ChatGPT Super App

By Akmal Chaudhri

CORE

SingleStore is a powerful multi-model database system and platform designed to support a wide variety of business use cases. Its distinctive features allow businesses to unify multiple database systems into a single platform, reducing the Total Cost of Ownership (TCO) and simplifying developer workflows by eliminating the need for complex integration tools. In this article, we'll explore how SingleStore can transform email campaigns for a web analytics company, enabling the creation of personalized and highly targeted email content. The notebook file used in the article is available on GitHub. Introduction A web analytics company relies on email campaigns to engage with customers. However, a generic approach to targeting customers often misses opportunities to maximize business potential. A more effective solution would involve using a large language model (LLM) to craft personalized email messages. Consider a scenario where user behavior data are stored in a NoSQL database like MongoDB, while valuable documentation resides in a vector database, such as Pinecone. Managing these multiple systems can become complex and resource-intensive, highlighting the need for a unified solution. SingleStore, a versatile multi-model database, supports various data formats, including JSON, and offers built-in vector functions. It seamlessly integrates with LLMs, making it a powerful alternative to managing multiple database systems. In this article, we'll demonstrate how easily SingleStore can replace both MongoDB and Pinecone, simplifying operations without compromising functionality. In our example application, we'll use an LLM to generate unique emails for our customers. To help the LLM learn how to target our customers, we'll use a number of well-known analytics companies as learning material for the LLM. We'll further customize the content based on user behavior. Customer data are stored in MongoDB. Different stages of user behavior are stored in Pinecone. The user behavior will allow the LLM to generate personalized emails. Finally, we'll consolidate the data stored in MongoDB and Pinecone by using SingleStore. Create a SingleStore Cloud Account A previous article showed the steps to create a free SingleStore Cloud account. We'll use the Standard Tier and take the default names for the Workspace Group and Workspace. We'll also enable SingleStore Kai. We'll store our OpenAI API Key and Pinecone API Key in the secrets vault using OPENAI_API_KEY and PINECONE_API_KEY, respectively. Import the Notebook We'll download the notebook from GitHub. From the left navigation pane in the SingleStore cloud portal, we'll select "DEVELOP" > "Data Studio." In the top right of the web page, we'll select "New Notebook" > "Import From File." We'll use the wizard to locate and import the notebook we downloaded from GitHub. Run the Notebook Generic Email Template We'll start by generating generic email templates and then use an LLM to transform them into personalized messages for each customer. This way, we can address each recipient by name and introduce them to the benefits of our web analytics platform. We can generate a generic email as follows: Python people = ["Alice", "Bob", "Charlie", "David", "Emma"] for person in people: message = ( f"Hey {person},\n" "Check out our web analytics platform, it's Awesome!\n" "It's perfect for your needs. Buy it now!\n" "- Marketer John" ) print(message) print("_" * 100) For example, Alice would see the following message: Plain Text Hey Alice, Check out our web analytics platform, it's Awesome! It's perfect for your needs. Buy it now! - Marketer John Other users would receive the same message, but with their name, respectively. 2. Adding a Large Language Model (LLM) We can easily bring an LLM into our application by providing it with a role and giving it some information, as follows: Python system_message = """ You are a helpful assistant. My name is Marketer John. You help write the body of an email for a fictitious company called 'Awesome Web Analytics'. This is a web analytics company that is similar to the top 5 web analytics companies (perform a web search to determine the current top 5 web analytics companies). The goal is to write a custom email to users to get them interested in our services. The email should be less than 150 words. Address the user by name. End with my signature. """ We'll create a function to call the LLM: Python def chatgpt_generate_email(prompt, person): conversation = [ {"role": "system", "content": prompt}, {"role": "user", "content": person}, {"role": "assistant", "content": ""} ] response = openai_client.chat.completions.create( model = "gpt-4o-mini", messages = conversation, temperature = 1.0, max_tokens = 800, top_p = 1, frequency_penalty = 0, presence_penalty = 0 ) assistant_reply = response.choices[0].message.content return assistant_reply Looping through the list of users and calling the LLM produces unique emails: Python openai_client = OpenAI() # Define a list to store the responses emails = [] # Loop through each person and generate the conversation for person in people: email = chatgpt_generate_email(system_message, person) emails.append( { "person": person, "assistant_reply": email } ) For example, this is what Alice might see: Plain Text Person: Alice Subject: Unlock Your Website's Potential with Awesome Web Analytics! Hi Alice, Are you ready to take your website to new heights? At Awesome Web Analytics, we provide cutting-edge insights that empower you to make informed decisions and drive growth. With our powerful analytics tools, you can understand user behavior, optimize performance, and boost conversions—all in real-time! Unlike other analytics platforms, we offer personalized support to guide you every step of the way. Join countless satisfied customers who have transformed their online presence. Discover how we stack up against competitors like Google Analytics, Adobe Analytics, and Matomo, but with a focus on simplicity and usability. Let us help you turn data into your greatest asset! Best, Marketer John Awesome Web Analytics Equally unique emails will be generated for the other users. 3. Customizing Email Content With User Behavior By categorising users based on their behavior stages, we can further customize email content to align with their specific needs. An LLM will assist in crafting emails that encourage users to progress through different stages, ultimately improving their understanding and usage of various services. At present, user data are held in a MongoDB database with a record structure similar to the following: JSON { '_id': ObjectId('64afb3fda9295d8421e7a19f'), 'first_name': 'James', 'last_name': 'Villanueva', 'company_name': 'Foley-Turner', 'stage': 'generating a tracking code', 'created_date': 1987-11-09T12:43:26.000+00:00 } We'll connect to MongoDB to get the data as follows: Python try: mongo_client = MongoClient("mongodb+srv://admin:<password>@<host>/?retryWrites=true&w=majority") mongo_db = mongo_client["mktg_email_demo"] collection = mongo_db["customers"] print("Connected successfully") except Exception as e: print(e) We'll replace <password> and <host> with the values from MongoDB Atlas. We have a number of user behavior stages: Python stages = [ "getting started", "generating a tracking code", "adding tracking to your website", "real-time analytics", "conversion tracking", "funnels", "user segmentation", "custom event tracking", "data export", "dashboard customization" ] def find_next_stage(current_stage): current_index = stages.index(current_stage) if current_index < len(stages) - 1: return stages[current_index + 1] else: return stages[current_index] Using the data about behavior stages, we'll ask the LLM to further customize the email as follows: Python limit = 5 emails = [] for record in collection.find(limit = limit): fname, stage = record.get("first_name"), record.get("stage") next_stage = find_next_stage(stage) system_message = f""" You are a helpful assistant, who works for me, Marketer John at Awesome Web Analytics. You help write the body of an email for a fictitious company called 'Awesome Web Analytics'. We are a web analytics company similar to the top 5 web analytics companies. We have users at various stages in our product's pipeline, and we want to send them helpful emails to encourage further usage of our product. Please write an email for {fname} who is on stage {stage} of the onboarding process. The next stage is {next_stage}. Ensure the email describes the benefits of moving to the next stage. Limit the email to 1 paragraph. End the email with my signature. """ email = chatgpt_generate_email(system_message, fname) emails.append( { "fname": fname, "stage": stage, "next_stage": next_stage, "email": email } ) For example, here is an email generated for Michael: Plain Text First Name: Michael Stage: funnels Next Stage: user segmentation Subject: Unlock Deeper Insights with User Segmentation! Hi Michael, Congratulations on successfully navigating the funnel stage of our onboarding process! As you move forward to user segmentation, you'll discover how this powerful tool will enable you to categorize your users based on their behaviors and demographics. By understanding your audience segments better, you can create tailored experiences that increase engagement and optimize conversions. This targeted approach not only enhances your marketing strategies but also drives meaningful results and growth for your business. We're excited to see how segmentation will elevate your analytics efforts! Best, Marketer John Awesome Web Analytics 4. Further Customizing Email Content To support user progress, we'll use Pinecone's vector embeddings, allowing us to direct users to relevant documentation for each stage. These embeddings make it effortless to guide users toward essential resources and further enhance their interactions with our product. Python pc = Pinecone( api_key = pc_api_key ) index_name = "mktg-email-demo" if any(index["name"] == index_name for index in pc.list_indexes()): pc.delete_index(index_name) pc.create_index( name = index_name, dimension = dimensions, metric = "euclidean", spec = ServerlessSpec( cloud = "aws", region = "us-east-1" ) ) pc_index = pc.Index(index_name) pc.list_indexes() We'll create the embeddings as follows: Python def get_embeddings(text): text = text.replace("\n", " ") try: response = openai_client.embeddings.create( input = text, model = "text-embedding-3-small" ) return response.data[0].embedding, response.usage.total_tokens, "success" except Exception as e: print(e) return "", 0, "failed" id_counter = 1 ids_list = [] for stage in stages: embedding, tokens, status = get_embeddings(stage) parent = id_counter - 1 pc_index.upsert([ { "id": str(id_counter), "values": embedding, "metadata": {"content": stage, "parent": str(parent)} } ]) ids_list.append(str(id_counter)) id_counter += 1 We'll search Pinecone for matches as follows: Python def search_pinecone(embedding): match = pc_index.query( vector = [embedding], top_k = 1, include_metadata = True )["matches"][0]["metadata"] return match["content"], match["parent"] Using the data, we can ask the LLM to further customize the email, as follows: Python limit = 5 emails = [] for record in collection.find(limit = limit): fname, stage = record.get("first_name"), record.get("stage") # Get the current and next stages with their embedding this_stage = next((item for item in stages_w_embed if item["stage"] == stage), None) next_stage = next((item for item in stages_w_embed if item["stage"] == find_next_stage(stage)), None) if not this_stage or not next_stage: continue # Get content cur_content, cur_permalink = search_pinecone(this_stage["embedding"]) next_content, next_permalink = search_pinecone(next_stage["embedding"]) system_message = f""" You are a helpful assistant. I am Marketer John at Awesome Web Analytics. We are similar to the current top web analytics companies. We have users at various stages of using our product, and we want to send them helpful emails to encourage them to use our product more. Write an email for {fname}, who is on stage {stage} of the onboarding process. The next stage is {next_stage['stage']}. Ensure the email describes the benefits of moving to the next stage, and include this link: https://github.com/VeryFatBoy/mktg-email-flow/tree/main/docs/{next_content.replace(' ', '-')}.md. Limit the email to 1 paragraph. End the email with my signature: 'Best Regards, Marketer John.' """ email = chatgpt_generate_email(system_message, fname) emails.append( { "fname": fname, "stage": stage, "next_stage": next_stage["stage"], "email": email } ) For example, here is an email generated for Melissa: Plain Text First Name: Melissa Stage: getting started Next Stage: generating a tracking code Subject: Take the Next Step with Awesome Web Analytics! Hi Melissa, We're thrilled to see you getting started on your journey with Awesome Web Analytics! The next step is generating your tracking code, which will allow you to start collecting valuable data about your website visitors. With this data, you can gain insights into user behavior, optimize your marketing strategies, and ultimately drive more conversions. To guide you through this process, check out our detailed instructions here: [Generating a Tracking Code](https://github.com/VeryFatBoy/mktg-email-flow/tree/main/docs/generating-a-tracking-code.md). We're here to support you every step of the way! Best Regards, Marketer John. We can see that we have refined the generic template and developed quite targeted emails. Using SingleStore Instead of managing separate database systems, we'll streamline our operations by using SingleStore. With its support for JSON, text, and vector embeddings, we can efficiently store all necessary data in one place, reducing TCO and simplifying our development processes. We'll ingest the data from MongoDB using a pipeline similar to the following: SQL USE mktg_email_demo; CREATE LINK mktg_email_demo.link AS MONGODB CONFIG '{"mongodb.hosts": "<primary>:27017, <secondary>:27017, <secondary>:27017", "collection.include.list": "mktg_email_demo.*", "mongodb.ssl.enabled": "true", "mongodb.authsource": "admin", "mongodb.members.auto.discover": "false"}' CREDENTIALS '{"mongodb.user": "admin", "mongodb.password": "<password>"}'; CREATE TABLES AS INFER PIPELINE AS LOAD DATA LINK mktg_email_demo.link '*' FORMAT AVRO; START ALL PIPELINES; We'll replace <primary>, <secondary>, <secondary> and <password> with the values from MongoDB Atlas. The customer table will be created by the pipeline. The vector embeddings for the behavior stages can be created as follows: Python df_list = [] id_counter = 1 for stage in stages: embedding, tokens, status = get_embeddings(stage) parent = id_counter - 1 stage_df = pd.DataFrame( { "id": [id_counter], "content": [stage], "embedding": [embedding], "parent": [parent] } ) df_list.append(stage_df) id_counter += 1 df = pd.concat(df_list, ignore_index = True) We'll need a table to store the data: SQL USE mktg_email_demo; DROP TABLE IF EXISTS docs_splits; CREATE TABLE IF NOT EXISTS docs_splits ( id INT, content TEXT, embedding VECTOR(:dimensions), parent INT ); Then, we can save the data in the table: Python df.to_sql( "docs_splits", con = db_connection, if_exists = "append", index = False, chunksize = 1000 ) We'll search SingleStore for matches as follows: Python def search_s2(vector): query = """ SELECT content, parent FROM docs_splits ORDER BY (embedding <-> :vector) ASC LIMIT 1 """ with db_connection.connect() as con: result = con.execute(text(query), {"vector": str(vector)}) return result.fetchone() Using the data, we can ask the LLM to customize the email as follows: Python limit = 5 emails = [] # Create a connection with db_connection.connect() as con: query = "SELECT _more :> JSON FROM customers LIMIT :limit" result = con.execute(text(query), {"limit": limit}) for customer in result: customer_data = customer[0] fname, stage = customer_data["first_name"], customer_data["stage"] # Retrieve current and next stage embeddings this_stage = next((item for item in stages_w_embed if item["stage"] == stage), None) next_stage = next((item for item in stages_w_embed if item["stage"] == find_next_stage(stage)), None) if not this_stage or not next_stage: continue # Get content cur_content, cur_permalink = search_s2(this_stage["embedding"]) next_content, next_permalink = search_s2(next_stage["embedding"]) # Create the system message system_message = f""" You are a helpful assistant. I am Marketer John at Awesome Web Analytics. We are similar to the current top web analytics companies. We have users that are at various stages in using our product, and we want to send them helpful emails to get them to use our product more. Write an email for {fname} who is on stage {stage} of the onboarding process. The next stage is {next_stage['stage']}. Ensure the email describes the benefits of moving to the next stage, then always share this link: https://github.com/VeryFatBoy/mktg-email-flow/tree/main/docs/{next_content.replace(' ', '-')}.md. Limit the email to 1 paragraph. End the email with my signature: 'Best Regards, Marketer John.' """ email = chatgpt_generate_email(system_message, fname) emails.append( { "fname": fname, "stage": stage, "next_stage": next_stage["stage"], "email": email, } ) For example, here is an email generated for Joseph: Plain Text First Name: Joseph Stage: generating a tracking code Next Stage: adding tracking to your website Subject: Take the Next Step in Your Analytics Journey! Hi Joseph, Congratulations on generating your tracking code! The next step is to add tracking to your website, which is crucial for unlocking the full power of our analytics tools. By integrating the tracking code, you will start collecting valuable data about your visitors, enabling you to understand user behavior, optimize your website, and drive better results for your business. Ready to get started? Check out our detailed guide here: [Adding Tracking to Your Website](https://github.com/VeryFatBoy/mktg-email-flow/tree/main/docs/adding-tracking-to-your-website.md). Best Regards, Marketer John. Summary Through this practical demonstration, we've seen how SingleStore improves our email campaigns with its multi-model capabilities and AI-driven personalization. Using SingleStore as our single source of truth, we've simplified our workflows and ensured that our email campaigns deliver maximum impact and value to our customers. Acknowledgements I thank Wes Kennedy for the original demo code, which was adapted for this article. More

Have You Heard About Cloud Native Buildpacks?

By Alvin Lee

CORE

How to Use GPU Nodes in Amazon EKS

By Alexander Sharov

Floyd's Cycle Algorithm for Fraud Detection in Java Systems

By Sulakshana Singh

A Glimpse Into the Future for Developers and Leaders

Software development is on the cusp of major transformations, driven by new technologies and an ever-growing demand for faster, more efficient, and scalable systems. For developers and leaders in software engineering, staying ahead of these trends will be essential to delivering cutting-edge solutions and keeping teams competitive. Let’s dive into some of the key software development trends that will define the industry in the near future. AI and ML: Setting the Trend Artificial intelligence (AI) and machine learning (ML) have already begun to impact how software is built, and in 2025, their role will become even more central. AI-powered tools like GitHub Copilot are already helping developers write code more quickly by suggesting improvements and detecting bugs. With ML, software can optimize its own performance based on patterns and data collected over time. These tools will become more sophisticated, significantly increasing developer productivity and ensuring higher-quality software development. For leaders, embracing AI-driven tools means not just improving efficiency but also fostering a culture of continuous learning and innovation. AI will serve as a powerful ally, amplifying human creativity and problem-solving skills. Ethical AI: Building Fair and Transparent Systems With the rapid growth of AI, there will be a greater emphasis on ethical AI practices. Developers and organizations will need to ensure that their AI systems are transparent, fair, and unbiased. Ethical concerns around data privacy, algorithmic fairness, and accountability will be at the forefront. And setting the ethical boundaries becomes key in ensuring adoption. Developers will be required to focus on building AI models that adhere to ethical guidelines, and leaders will need to foster a culture where AI is developed responsibly. This may involve implementing bias detection and mitigation strategies and ensuring compliance with evolving regulations surrounding AI technologies. AI-Powered Code Reviews: Speed, Consistency, and Quality AI can assist developers by automating code reviews. ML models can analyze code for potential bugs, inefficiencies, or adherence to best practices. These tools provide real-time suggestions for optimization and error detection, leading to quicker iterations and better-quality software. AI-powered code reviews can help reduce human bias, uncover edge cases, and offer consistent, objective feedback. For leaders, adopting AI-powered code reviews means ensuring consistency and quality across development teams. Cloud Native Development and Microservices Stay Relevant Cloud-native development continues to dominate the way companies build scalable software solutions. Organizations continue to increasingly migrate to the cloud, using cloud platforms like AWS, Google Cloud, and Azure. The shift will also bring a rise in microservices, where applications are broken down into smaller, independent services that can be updated or scaled without affecting the entire system. For developers, mastering cloud tools and technologies like Docker, Kubernetes, and containerization will be essential. From a leadership perspective, adopting cloud-native solutions provides more flexibility and scalability along with allowing teams to go at a faster pace. Serverless Computing: More Flexibility With Less Effort Serverless computing is set to become even more popular. Traditional architectures require developers to manage servers, but serverless platforms like AWS Lambda and Google Cloud Functions handle infrastructure for you. This means developers can focus on writing code while the cloud provider takes care of the resources. Serverless computing is especially useful for applications with unpredictable traffic, allowing resources to scale up or down based on demand. With serverless, from a cost perspective, businesses only pay for the computing power they actually use, making it a more affordable option for many. And this is one of the ways leaders can work towards cost savings in their organization. CI/CD: Automation All the Way DevOps, along with continuous integration and continuous deployment, is already revolutionizing how software is developed and released. These practices will become even more integral to ensuring fast and reliable software delivery. CI/CD pipelines automate the testing and deployment of software, allowing developers to push out new features, fixes, and updates with greater speed. For developers, mastering CI/CD tools like Jenkins and GitLab will be crucial, while leaders will need to ensure that teams work in a collaborative environment that emphasizes rapid, continuous development and deployment. Security With DevSecOps: Shift Towards Secure Development As cyber threats continue to increase, security is becoming a more prominent concern in software development. Security is no longer a separate concern handled by a different team. Instead, security will be integrated into the development process, ensuring that every stage of development includes security checks. Developers will need to adopt secure coding practices and integrate security tools directly into their CI/CD pipelines. Leaders must create a culture where security is seen as everyone’s responsibility, not just something that is handled after the software is built. Quantum Computing: Get to Know It Quantum computing is still in its infancy, but it will continue to advance significantly. Though quantum computers will not replace traditional computers, they have the potential to revolutionize fields like cryptography, optimization, and large-scale simulations. As this technology progresses, software engineers will need to prepare for the new challenges and opportunities that quantum computing will bring. Software engineers will need to familiarize themselves with quantum algorithms and specialized programming languages to leverage this emerging technology. For leaders, it’s time to start exploring how quantum computing might fit into long-term strategies for solving complex problems. Architecture Pattern: Blueprint for Scalable Systems Architecture patterns are essential for ensuring that software systems are scalable, efficient, and maintainable. There continues to be an increased reliance on them to handle the complexity of modern applications. Microservices, for instance, will continue to be a dominant pattern. By breaking down monolithic systems into smaller, independently deployable services, microservices allow development teams to work on different components of a system simultaneously. This approach fosters agility, quick scaling, and more manageable development cycles.Event-driven architecture is another pattern that allows systems to respond to events, providing more flexibility in handling real-time data and improving system responsiveness. Domain-driven design (DDD) will help teams organize software projects by focusing on the business domain and ensuring that software models match real-world processes. DDD will allow developers to design systems that are adaptable and aligned with business goals.AI-driven design patterns are patterns that leverage AI to optimize and automate various aspects of system architecture. These patterns focus on creating systems that can learn and evolve based on data, automate decision-making processes, and learn from past interactions. Examples include intelligent routing patterns, predictive analytics models, and AI-enhanced event-driven architectures. Leaders must be willing to adopt fundamental shifts to ensure systems are built for flexibility, scalability, and long-term sustainability. Conclusion The software development landscape is shaping up for an incredible change. From AI and cloud-native development to serverless architectures and ethical AI, the trends emerging now will redefine how developers and engineering leaders approach their work. To stay competitive, developers must continuously learn new technologies and adopt best practices like security-first mindsets. Engineering leaders, on the other hand, must foster innovation, encourage ethical practices, and ensure that their teams are equipped with the skills and tools to thrive in the evolving tech landscape.

By Sangame Krishnamani

Automating Podcast Promotion With AI and Event-Driven Design

I host two podcasts, Software Engineering Daily and Software Huddle, and often appear as a guest on other shows. Promoting episodes — whether I’m hosting or featured — helps highlight the great conversations I have, but finding the time to craft a thoughtful LinkedIn post for each one is tough. Between hosting, work, and life, sitting down to craft a thoughtful LinkedIn post for every episode just doesn’t always happen. To make life easier (and still do justice to the episodes), I built an AI-powered LinkedIn post generator. It downloads podcast episodes, converts the audio into text, and uses that to create posts. It saves me time, keeps my content consistent, and ensures each episode gets the spotlight it deserves. In this post, I break down how I built this tool using Next.js, OpenAI’s GPT and Whisper models, Apache Kafka, and Apache Flink. More importantly, I’ll show how Kafka and Flink power an event-driven architecture, making it scalable and reactive — a pattern critical for real-time AI applications. Note: If you would like to just look at the code, jump over to my GitHub repo here. Designing a LinkedIn Post Assistant The goal of this application was pretty straightforward: help me create LinkedIn posts for the podcasts I’ve hosted or guested on without eating up too much of my time. To meet my needs, I wanted to be able to provide a URL for a podcast feed, pull in the list of all episodes, and then generate a LinkedIn post for any episode I chose. Simple, right? Of course, there’s some heavy lifting under the hood to make it all work: Download the MP3 for the selected episode.Convert the audio to text using OpenAI’s Whisper model.Since Whisper has a 25MB file size limit, split the MP3 into smaller chunks if needed.Finally, use the transcription to assemble a prompt and ask an LLM to generate the LinkedIn post for me. Beyond functionality, I also had another important goal: keep the front-end app completely decoupled from the AI workflow. Why? Because in real-world AI applications, teams usually handle different parts of the stack. Your front-end developers shouldn’t need to know anything about AI to build the user-facing app. Plus, I wanted the flexibility to: Scale different parts of the system independently.Swap out models or frameworks as the ever-growing generative AI stack evolves. To achieve all this, I implemented an event-driven architecture using Confluent Cloud. This approach not only keeps things modular but also sets the stage for future-proofing the application as AI technologies inevitably change. Why an Event-Driven Architecture for AI? Event-driven architecture (EDA) emerged as a response to the limitations of traditional, monolithic systems that relied on rigid, synchronous communication patterns. In the early days of computing, applications were built around static workflows, often tied to batch processes or tightly coupled interactions. The architecture of a single server monolith As technology evolved and the demand for scalability and adaptability grew — especially with the rise of distributed systems and microservices — EDA became a natural solution. By treating events — such as state changes, user actions, or system triggers — as the core unit of interaction, EDA enables systems to decouple components and communicate asynchronously. This approach uses data streaming, where producers and consumers interact through a shared, immutable log. Events are persisted in a guaranteed order, allowing systems to process and react to changes dynamically and independently. High-level overview of event producers and consumers Decoupling My Web App from the AI Workflow Bringing this back to the task at hand, my web application doesn’t need to know anything about AI. To decouple the user-facing application from the AI, I used Confluent Cloud’s data streaming platform, which supports Kafka, Flink, and AI models as first-class citizens, making it easy to build a truly scalable AI application. When a user clicks on a podcast listing, the app asks the server to check a backend cache for an existing LinkedIn post. If one is found, it’s returned and displayed. You could store these LinkedIn posts in a database, but I chose a temporary cache since I don’t really need to persist these for very long. If no LinkedIn post exists, the backend writes an event to a Kafka topic, including the MP3 URL and episode description. This triggers the workflow to generate the LinkedIn post. The diagram below illustrates the full architecture of this event-driven system, which I explain in more detail in the next section. An event-driven workflow for AI-generated LinkedIn posts Downloading and Generating the Transcript This part of the workflow is fairly straightforward. The web app writes requests to a Kafka topic named LinkedIn Post Request. Using Confluent Cloud, I configured an HTTP Sink Connector to forward new messages to an API endpoint. The API endpoint downloads the MP3 using the provided URL, splits the file into 25 MB chunks if necessary, and processes the audio with Whisper to generate a transcript. As transcriptions are completed, they are written to another Kafka topic called Podcast Transcript. This is where the workflow gets interesting — stream processing begins to handle the heavy lifting. Generating the LinkedIn Post Apache Flink is an open-source stream-processing framework designed to handle large volumes of data in real time. It excels in high-throughput, low-latency scenarios, making it a great fit for real-time AI applications. If you're familiar with databases, you can think of Flink SQL as similar to standard SQL, but instead of querying database tables, you query a data stream. To use Flink for turning podcast episodes into LinkedIn posts, I needed to integrate an external LLM. Flink SQL makes this straightforward by allowing you to define a model for widely used LLMs. You can specify the task (e.g., text_generation) and provide a system prompt to guide the output, as shown below. SQL CREATE MODEL `linkedin_post_generation` INPUT (text STRING) OUTPUT (response STRING) WITH ( 'openai.connection'='openai-connection', 'provider'='openai', 'task'='text_generation', 'openai.model_version' = 'gpt-4', 'openai.system_prompt' = 'You are an expert in AI, databases, and data engineering. You need to write a LinkedIn post based on the following podcast transcription and description. The post should summarize the key points, be concise, direct, free of jargon, but thought-provoking. The post should demonstrate a deep understanding of the material, adding your own takes on the material. Speak plainly and avoid language that might feel like a marketing person wrote it. Avoid words like "delve", "thought-provoking". Make sure to mention the guest by name and the company they work for. Keep the tone professional and engaging, and tailor the post to a technical audience. Use emojis sparingly.' ); To create the LinkedIn post, I first joined the LinkedIn Post Request topic and Podcast Transcript topic based on the MP3 URL to combine the episode description and transcript into a prompt value that I store in a view. Using a view improves readability and maintainability; while I could have embedded the string concatenation directly in the ml_predict call, doing so would make the workflow harder to modify. SQL CREATE VIEW podcast_prompt AS SELECT mp3.key AS key, mp3.mp3Url AS mp3Url, CONCAT( 'Generate a concise LinkedIn post that highlights the main points of the podcast while mentioning the guest and their company.', CHR(13), CHR(13), 'Podcast Description:', CHR(13), rqst.episodeDescription, CHR(13), CHR(13), 'Podcast Transcript:', CHR(13), mp3.transcriptionText ) AS prompt FROM `linkedin-podcast-mp3` AS mp3 JOIN `linkedin-generation-request` AS rqst ON mp3.mp3Url = rqst.mp3Url WHERE mp3.transcriptionText IS NOT NULL; Once the prompts are prepared in the view, I use another Flink SQL statement to generate the LinkedIn post by passing the prompt to the LLM model that I set up previously. The completed post is then written to a new Kafka topic, Completed LinkedIn Posts. This approach simplifies the process while keeping the workflow scalable and flexible. SQL INSERT INTO `linkedin-request-complete` SELECT podcast.key, podcast.mp3Url, prediction.response FROM `podcast_prompt` AS podcast CROSS JOIN LATERAL TABLE ( ml_predict( 'linkedin_post_generation', podcast.prompt ) ) AS prediction; Writing the Post to the Cache The final step is configuring another HTTP Sink Connector in Confluent Cloud to send the completed LinkedIn post to an API endpoint. This endpoint writes the data to the backend cache. Once cached, the LinkedIn post becomes available to the front-end application, which automatically displays the result as soon as it’s ready. Key Takeaways Building an AI-powered LinkedIn post generator was more than just a way to save time — it was an exercise in designing a modern, scalable, and decoupled event-driven system. As with any software project, choosing the right architecture upfront is vitally important. The generative AI landscape is evolving rapidly, with new models, frameworks, and tools emerging constantly. By decoupling components and embracing event-driven design, you can future-proof your system, making it easier to adopt new technologies without overhauling your entire stack. Decouple your workflows, embrace event-driven systems, and ensure your architecture allows for seamless scaling and adaptation. Whether you’re building a LinkedIn post generator or tackling more complex AI use cases, these principles are universal. If this project resonates with you, feel free to explore the code on GitHub or reach out to me on LinkedIn to discuss it further. Happy building!

By Sean Falconer

Dynamic Forms With Camunda and Spring StateMachine

In modern process automation, flexibility and adaptability are key. Processes often require dynamic forms that can change based on user input, business rules, or external factors. Traditional approaches, where forms are hardcoded into the process definition, can be rigid and difficult to maintain. This article presents a flexible and scalable approach to handling dynamic forms in process automation, using Camunda BPM and Spring StateMachine as the underlying engines. We’ll explore how to decouple form definitions from process logic, enabling dynamic form rendering, validation, and submission. This approach is applicable to both Camunda and Spring StateMachine, making it a versatile solution for various process automation needs. The Problem: Static Forms in Process Automation In traditional process automation systems, forms are often tied directly to tasks in the process definition. For example, in Camunda, the camunda:formKey attribute is used to associate a form with a task. While this works for simple processes, it has several limitations: Rigidity. Forms are hardcoded into the process definition, making it difficult to modify them without redeploying the process.Lack of flexibility. Forms cannot easily adapt to dynamic conditions, such as user roles, input data, or business rules.Maintenance overhead. Changes to forms require updates to the process definition, leading to increased maintenance complexity. To address these limitations, we often need a more dynamic and decoupled approach to form handling. The Solution: Dynamic Forms With YAML Configuration Our solution involves decoupling form definitions from the process logic by storing form configurations in a YAML file (or another external source). This allows forms to be dynamically served based on the current state of the process, user roles, or other runtime conditions. Key Components 1. Process Engine Camunda BPM – A powerful BPMN-based process engine for complex workflows.Spring State Machine – A lightweight alternative for simpler state-based processes. 2. Form Configuration Forms are defined in a YAML file, which includes fields, labels, types, validation rules, and actions. 3. Backend Service A Spring Boot service that serves form definitions dynamically based on the current process state. 4. Frontend A dynamic frontend (e.g., React, Angular, or plain HTML/JavaScript) that renders forms based on the configuration returned by the backend. Implementation With Camunda BPM The complete source code of this approach is at the camunda branch of the spring-statemachine-webapp GitHub repository. Let's dive into it: 1. BPMN Model The BPMN model defines the process flow without any reference to forms. For example: XML <process id="loanApplicationProcess_v1" name="Loan Application Process" isExecutable="true"> <startEvent id="startEvent" name="Start"/> <sequenceFlow id="flow1" sourceRef="startEvent" targetRef="personalInformation"/> <userTask id="personalInformation" name="Personal Information"/> <sequenceFlow id="flow2" sourceRef="personalInformation" targetRef="loanDetails"/>  </process> The BPMN diagram is depicted below: 2. YAML Form Configuration Forms are defined in a YAML file, making them easy to modify and extend: YAML processes: loan_application: steps: personalInformation: title: "Personal Information" fields: - id: "firstName" label: "First Name" type: "text" required: true - id: "lastName" label: "Last Name" type: "text" required: true actions: - id: "next" label: "Next" event: "STEP_ONE_SUBMIT" 3. Backend Service The backend service (ProcessService.java) dynamically handles the submission of steps, the persistence of form data, and of course, it serves form definitions based on the current task: Java @Transactional(readOnly = true) public Map<String, Object> getFormDefinition(String processId) { Task task = taskService.createTaskQuery().processInstanceBusinessKey(processId).singleResult(); //... String processType = (String) runtimeService.getVariable(task.getProcessInstanceId(), "type"); FormFieldConfig.ProcessConfig processConfig = formFieldConfig.getProcesses().get(processType); // ... String stepKey = task.getTaskDefinitionKey(); FormFieldConfig.StepConfig stepConfig = processConfig.getSteps().get(stepKey); //... Map<String, Object> result = new HashMap<>(); result.put("processId", processId); result.put("processType", processType); result.put("currentState", stepKey); // Using task definition key as the current state result.put("step", stepKey); result.put("title", stepConfig.getTitle()); result.put("fields", stepConfig.getFields()); result.put("actions", stepConfig.getActions()); List<FormData> previousData = formDataRepository.findByProcessIdAndStep(processId, stepKey); if (!previousData.isEmpty()) { result.put("data", previousData.get(0).getFormDataJson()); } return result; } 4. Frontend The frontend dynamically renders forms based on the configuration returned by the backend: JavaScript function loadStep() { // Fetch form configuration from the backend $.get(`/api/process/${processId}/form`, function(data) { $("#formContainer").empty(); if (data.currentState === "submission") { // Fetch and render the process summary... } else { // Render the form dynamically let formHtml = `<h3>${data.title}</h3><form id="stepForm">`; // Render form fields (text, number, date, select, etc.) data.fields.forEach(field => { //... }); // Render form actions (e.g., Next, Back, Submit) formHtml += `<div>`; data.actions.forEach(action => { formHtml += `<button onclick="submitStep('${data.step}', '${action.event}')">${action.label}</button>`; }); formHtml += `</div></form>`; $("#formContainer").html(formHtml); // Initialize date pickers and restore previous data... } }).fail(function() { $("#formContainer").html(`<p>Error loading form. Please try again.</p>`); }); } Implementation With Spring StateMachine The complete source code of this approach is at the main branch of the spring-statemachine-webapp GitHub repository. There is also another flavor with Spring StateMachine persistence enabled in the branch enable-state-machine-persistence. 1. State Machine Configuration The state machine defines the process flow without any reference to forms: Java @Configuration @EnableStateMachineFactory public class StateMachineConfig extends EnumStateMachineConfigurerAdapter<ProcessStates, ProcessEvents> { //... @Override public void configure(StateMachineStateConfigurer<ProcessStates, ProcessEvents> states) throws Exception { states .withStates() .initial(ProcessStates.PROCESS_SELECTION) .states(EnumSet.allOf(ProcessStates.class)) .end(ProcessStates.COMPLETED) .end(ProcessStates.ERROR); } @Override public void configure(StateMachineTransitionConfigurer<ProcessStates, ProcessEvents> transitions) throws Exception { transitions .withExternal() .source(ProcessStates.PROCESS_SELECTION) .target(ProcessStates.STEP_ONE) .event(ProcessEvents.PROCESS_SELECTED) .action(saveProcessAction()) .and() .withExternal() .source(ProcessStates.STEP_ONE) .target(ProcessStates.STEP_TWO) .event(ProcessEvents.STEP_ONE_SUBMIT) .and() .withExternal() .source(ProcessStates.STEP_TWO) .target(ProcessStates.STEP_THREE) .event(ProcessEvents.STEP_TWO_SUBMIT) .and() .withExternal() .source(ProcessStates.STEP_THREE) .target(ProcessStates.SUBMISSION) .event(ProcessEvents.STEP_THREE_SUBMIT) .and() .withExternal() .source(ProcessStates.SUBMISSION) .target(ProcessStates.COMPLETED) .event(ProcessEvents.FINAL_SUBMIT) .and() // Add back navigation .withExternal() .source(ProcessStates.STEP_TWO) .target(ProcessStates.STEP_ONE) .event(ProcessEvents.BACK) .and() .withExternal() .source(ProcessStates.STEP_THREE) .target(ProcessStates.STEP_TWO) .event(ProcessEvents.BACK) .and() .withExternal() .source(ProcessStates.SUBMISSION) .target(ProcessStates.STEP_THREE) .event(ProcessEvents.BACK) } //... } The state machine diagram is depicted below: 2. YAML Form Configuration The same YAML file is used to define forms for both Camunda and Spring StateMachine. 3. Backend Service The backend service (ProcessService.java) is quite similar to the Camunda version, i.e, it has the same responsibilities and methods. The key differences here have to do with interacting with a state machine instead of a BPMN engine. For example, when we want to get the form definitions, the approach is like the snippet below: Java @Transactional(readOnly = true) public Map<String, Object> getFormDefinition(String processId) { StateMachineContext<ProcessStates, ProcessEvents> stateMachineContext = loadProcessContext(processId); String processType = (String) stateMachineContext.getExtendedState().getVariables().get("type"); String stepKey = stateToStepKey(stateMachineContext.getState()); FormFieldConfig.ProcessConfig processConfig = formFieldConfig.getProcesses().get(processType); // ... FormFieldConfig.StepConfig stepConfig = processConfig.getSteps().get(stepKey); //... Map<String, Object> result = new HashMap<>(); result.put("processId", processId); result.put("processType", processType); result.put("currentState", stateMachineContext.getState()); result.put("step", stepKey); result.put("title", stepConfig.getTitle()); result.put("fields", stepConfig.getFields()); result.put("actions", stepConfig.getActions()); // Add previously saved data if available List<FormData> previousData = formDataRepository.findByProcessIdAndStep(processId, stepKey); if (!previousData.isEmpty()) { result.put("data", previousData.get(0).getFormDataJson()); } return result; } 4. Frontend The frontend remains the same, dynamically rendering forms based on the configuration returned by the backend. Benefits of the Dynamic Forms Approach Flexibility. Forms can be modified without redeploying the process definition.Maintainability. Form definitions are centralized in a YAML file, making them easy to update.Scalability. The approach works for both simple and complex processes.Reusability. The same form configuration can be used across multiple processes. Conclusion We can create flexible, maintainable, and scalable process automation systems by decoupling form definitions from process logic. This approach works seamlessly with Camunda BPM and Spring StateMachine, making it a versatile solution for a wide range of use cases, whether building complex workflows or simple state-based processes. These dynamic forms can help you adapt to changing business requirements with ease.

By Kyriakos Mandalas

CORE

Go Flags: Beyond the Basics

The flag package is one of Go's most powerful standard library tools for building command-line applications. Understanding flags is essential whether you're creating a simple CLI tool or a complex application. Let's look into what makes this package so versatile. Basic Flag Concepts Let's start with a simple example that demonstrates the core concepts: Go package main import ( "flag" "fmt" ) func main() { // Basic flag definitions name := flag.String("name", "guest", "your name") age := flag.Int("age", 0, "your age") isVerbose := flag.Bool("verbose", false, "enable verbose output") // Parse the flags flag.Parse() // Use the flags fmt.Printf("Hello %s (age: %d)!\n", *name, *age) if *isVerbose { fmt.Println("Verbose mode enabled") } } Advanced Usage Patterns Custom Flag Types Sometimes, you need flags that aren't just simple types. Here's how to create a custom flag type: Go type IntervalFlag struct { Duration time.Duration } func (i *IntervalFlag) String() string { return i.Duration.String() } func (i *IntervalFlag) Set(value string) error { duration, err := time.ParseDuration(value) if err != nil { return err } i.Duration = duration return nil } func main() { interval := IntervalFlag{Duration: time.Second} flag.Var(&interval, "interval", "interval duration (e.g., 10s, 1m)") flag.Parse() } Using Flag Sets FlagSets are perfect for organizing flags in larger applications: Go package main import ( "flag" "fmt" "os" ) func main() { // Create flag sets for different commands serverCmd := flag.NewFlagSet("server", flag.ExitOnError) serverPort := serverCmd.Int("port", 8080, "server port") serverHost := serverCmd.String("host", "localhost", "server host") clientCmd := flag.NewFlagSet("client", flag.ExitOnError) clientTimeout := clientCmd.Duration("timeout", 30*time.Second, "client timeout") clientAddr := clientCmd.String("addr", "localhost:8080", "server address") if len(os.Args) < 2 { fmt.Println("expected 'server' or 'client' subcommand") os.Exit(1) } switch os.Args[1] { case "server": serverCmd.Parse(os.Args[2:]) runServer(*serverHost, *serverPort) case "client": clientCmd.Parse(os.Args[2:]) runClient(*clientAddr, *clientTimeout) default: fmt.Printf("unknown subcommand: %s\n", os.Args[1]) os.Exit(1) } } func runServer(host string, port int) { fmt.Printf("Running server on %s:%d\n", host, port) } func runClient(addr string, timeout time.Duration) { fmt.Printf("Connecting to %s with timeout %v\n", addr, timeout) } Real-World Example: Configuration Manager Here's a practical example of using flags to create a configuration management tool: Go package main import ( "flag" "fmt" "log" "os" "path/filepath" ) type Config struct { ConfigPath string LogLevel string Port int Debug bool DataDir string } func main() { config := parseFlags() if err := run(config); err != nil { log.Fatal(err) } } func parseFlags() *Config { config := &Config{} // Define flags with environment variable fallbacks flag.StringVar(&config.ConfigPath, "config", getEnvOrDefault("APP_CONFIG", "config.yaml"), "path to config file") flag.StringVar(&config.LogLevel, "log-level", getEnvOrDefault("APP_LOG_LEVEL", "info"), "logging level (debug, info, warn, error)") flag.IntVar(&config.Port, "port", getEnvIntOrDefault("APP_PORT", 8080), "server port number") flag.BoolVar(&config.Debug, "debug", getEnvBoolOrDefault("APP_DEBUG", false), "enable debug mode") flag.StringVar(&config.DataDir, "data-dir", getEnvOrDefault("APP_DATA_DIR", "./data"), "directory for data storage") // Custom usage message flag.Usage = func() { fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0]) flag.PrintDefaults() fmt.Fprintf(os.Stderr, "\nEnvironment variables:\n") fmt.Fprintf(os.Stderr, " APP_CONFIG Path to config file\n") fmt.Fprintf(os.Stderr, " APP_LOG_LEVEL Logging level\n") fmt.Fprintf(os.Stderr, " APP_PORT Server port\n") fmt.Fprintf(os.Stderr, " APP_DEBUG Debug mode\n") fmt.Fprintf(os.Stderr, " APP_DATA_DIR Data directory\n") } flag.Parse() // Validate configurations if err := validateConfig(config); err != nil { log.Fatalf("Invalid configuration: %v", err) } return config } func validateConfig(config *Config) error { // Check if config file exists if _, err := os.Stat(config.ConfigPath); err != nil { return fmt.Errorf("config file not found: %s", config.ConfigPath) } // Validate log level switch config.LogLevel { case "debug", "info", "warn", "error": // Valid log levels default: return fmt.Errorf("invalid log level: %s", config.LogLevel) } // Validate port range if config.Port < 1 || config.Port > 65535 { return fmt.Errorf("port must be between 1 and 65535") } // Ensure data directory exists or create it if err := os.MkdirAll(config.DataDir, 0755); err != nil { return fmt.Errorf("failed to create data directory: %v", err) } return nil } // Helper functions for environment variables func getEnvOrDefault(key, defaultValue string) string { if value, exists := os.LookupEnv(key); exists { return value } return defaultValue } func getEnvIntOrDefault(key string, defaultValue int) int { if value, exists := os.LookupEnv(key); exists { if parsed, err := strconv.Atoi(value); err == nil { return parsed } } return defaultValue } func getEnvBoolOrDefault(key string, defaultValue bool) bool { if value, exists := os.LookupEnv(key); exists { if parsed, err := strconv.ParseBool(value); err == nil { return parsed } } return defaultValue } func run(config *Config) error { log.Printf("Starting application with configuration:") log.Printf(" Config Path: %s", config.ConfigPath) log.Printf(" Log Level: %s", config.LogLevel) log.Printf(" Port: %d", config.Port) log.Printf(" Debug Mode: %v", config.Debug) log.Printf(" Data Directory: %s", config.DataDir) // Application logic here return nil } Best Practices and Tips 1. Default Values Always provide sensible default values for your flags. Go port := flag.Int("port", 8080, "port number (default 8080)") 2. Clear Descriptions Write clear, concise descriptions for each flag. Go flag.StringVar(&config, "config", "config.yaml", "path to configuration file") 3. Environment Variable Support Consider supporting both flags and environment variables. Go flag.StringVar(&logLevel, "log-level", os.Getenv("LOG_LEVEL"), "logging level") 4. Validation Always validate flag values after parsing. Go if *port < 1 || *port > 65535 { log.Fatal("Port must be between 1 and 65535") } 5. Custom Usage Provide clear usage information Go flag.Usage = func() { fmt.Fprintf(os.Stderr, "Usage of %s:\n", os.Args[0]) flag.PrintDefaults() } Advanced Patterns Combining With cobra While the flag package is powerful, you might want to combine it with more robust CLI frameworks like cobra: Go package main import ( "github.com/spf13/cobra" "github.com/spf13/viper" ) func main() { var rootCmd = &cobra.Command{ Use: "myapp", Short: "My application description", } // Add flags that can also be set via environment variables rootCmd.PersistentFlags().String("config", "", "config file (default is $HOME/.myapp.yaml)") viper.BindPFlag("config", rootCmd.PersistentFlags().Lookup("config")) if err := rootCmd.Execute(); err != nil { log.Fatal(err) } } Common Pitfalls to Avoid Not calling flag.Parse(). Always call flag.Parse() after defining your flags.Ignoring errors. Handle flag parsing errors appropriately.Using global flags. Prefer using FlagSets for better organization in larger applications.Missing documentation. Always provide clear documentation for all flags. Conclusion The flag package is a powerful tool for building command-line applications in Go. You can create robust and user-friendly CLI applications by understanding its advanced features and following best practices. Remember to validate inputs, provide clear documentation, and consider combining with other packages for more complex applications. Good luck with your CLI development!

By Rez Moss

KIAM vs AWS IAM Roles for Service Accounts (IRSA)

As Kubernetes adoption grows in cloud-native environments, securely managing AWS IAM roles within Kubernetes clusters has become a critical aspect of infrastructure management. KIAM and AWS IAM Roles for Service Accounts (IRSA) are two popular approaches to handling this requirement. In this article, we discuss the nuances of both tools, comparing their features, architecture, benefits, and drawbacks to help you make an informed decision for your Kubernetes environment. Introduction KIAM: An open-source solution designed to assign AWS IAM roles to Kubernetes pods dynamically, without storing AWS credentials in the pods themselves. KIAM uses a proxy-based architecture to intercept AWS metadata API requests.IRSA: AWS's official solution that leverages Kubernetes service accounts and OpenID Connect (OIDC) to securely associate IAM roles with Kubernetes pods. IRSA eliminates the need for an external proxy. Architecture and Workflow KIAM Components Agent – Runs as a DaemonSet on worker nodes, intercepting AWS metadata API calls from pods.Server – Centralized component handling IAM role validation and AWS API interactions. Workflow Pod metadata includes an IAM role annotation.The agent intercepts metadata API calls and forwards them to the server.The server validates the role and fetches temporary AWS credentials via STS.The agent injects the credentials into the pod’s metadata response. IRSA Components Kubernetes service accounts annotated with IAM role ARNs.An OIDC identity provider configured in AWS IAM. Workflow A service account is annotated with an IAM role.Pods that use the service account are issued a projected service account token.AWS STS validates the token via the OIDC identity provider.The pod assumes the associated IAM role. Feature Comparison Feature KIAM IRSA Setup Complexity Requires deploying KIAM components. Requires enabling OIDC and setting up annotations. Scalability Limited at scale due to proxy bottlenecks. Highly scalable; no proxy required. Maintenance Requires ongoing management of KIAM. Minimal maintenance; native AWS support. Security Credentials are fetched dynamically but flow through KIAM servers. Credentials are validated directly by AWS STS. Performance Metadata API interception adds latency. Direct integration with AWS; minimal latency. AWS Native Support No, third-party tool. Yes, fully AWS-supported solution. Multi-cloud Support No, AWS-specific. No, AWS-specific. Advantages and Disadvantages Advantages of KIAM Flexibility. Works in non-EKS Kubernetes clusters.Proven utility. Widely used before IRSA was introduced. Disadvantages of KIAM Performance bottlenecks. Metadata interception can lead to latency issues, especially in large-scale clusters.Scalability limitations. Centralized server can become a bottleneck.Security risks. Additional proxy layer increases the attack surface.Maintenance overhead. Requires managing and updating KIAM components. Advantages of IRSA AWS-native integration. Leverages native AWS features for seamless operation.Improved security. Credentials are issued directly via AWS STS without intermediaries.Better performance. No proxy overhead; direct STS interactions.Scalable. Ideal for large clusters due to its distributed nature. Disadvantages of IRSA AWS-only. Not suitable for multi-cloud or hybrid environments.Initial learning curve. Requires understanding OIDC and service account setup. Use Cases When to Use KIAM Non-EKS Kubernetes clusters.Scenarios where legacy systems rely on KIAM's specific functionality. When to Use IRSA EKS clusters or Kubernetes environments running on AWS.Use cases requiring scalability, high performance, and reduced maintenance overhead.Security-sensitive environments that demand minimal attack surface. Migration from KIAM to IRSA If you are currently using KIAM and want to migrate to IRSA, here’s a step-by-step approach: 1. Enable OIDC for Your Cluster In EKS, enable the OIDC provider using the AWS Management Console or CLI. 2. Annotate Service Accounts Replace IAM role annotations in pods with annotations in service accounts. 3. Update IAM Roles Add the OIDC identity provider to your IAM roles’ trust policy. 4. Test and Verify Deploy test workloads to ensure that the roles are assumed correctly via IRSA. 5. Decommission KIAM Gradually phase out KIAM components after successful migration. Best Practices for Migration Perform the migration incrementally, starting with non-critical workloads.Use a staging environment to validate changes before applying them to production.Monitor AWS CloudWatch metrics and logs to identify potential issues during the transition.Leverage automation tools like Terraform or AWS CDK to streamline the setup and configuration. Real-World Examples KIAM in Action Legacy systems – Organizations using non-EKS clusters where KIAM remains relevant due to its compatibility with diverse environmentsHybrid workloads – Enterprises running workloads across on-premise and cloud platforms IRSA Success Stories Modern applications – Startups leveraging IRSA for seamless scaling and enhanced security in AWS EKS environmentsEnterprise adoption – Large-scale Kubernetes clusters in enterprises benefiting from reduced maintenance overhead and native AWS integration Conclusion While KIAM was a groundbreaking tool in its time, AWS IAM Roles for Service Accounts (IRSA) has emerged as the preferred solution for managing IAM roles in Kubernetes environments running on AWS. IRSA offers native support, better performance, improved security, and scalability, making it a superior choice for modern cloud-native architectures. For Kubernetes clusters on AWS, IRSA should be the go-to option. However, if you operate outside AWS or in hybrid environments, KIAM or alternative tools may still have relevance. For infrastructure architects, DevOps engineers, and Kubernetes enthusiasts, this comparative analysis aims to provide the insights needed to choose the best solution for their environments. If you need deeper technical insights or practical guides, feel free to reach out.

By Kuppusamy Vellamadam Palavesam

Mastering Scalability in Spring Boot

Scalability is a fundamental concept in both technology and business that refers to the ability of a system, network, or organization to handle a growing amount of requests or ability to grow. This characteristic is crucial for maintaining performance and efficiency as need increases. In this article, we will explore the definition of scalability, its importance, types, methods to achieve it, and real-world examples. What Is Scalability in System Design? Scalability encompasses the capacity of a system to grow and manage increasing workloads without compromising performance. This means that as user traffic, data volume, or computational demands rise, a scalable system can maintain or even enhance its performance. The essence of scalability lies in its ability to adapt to growth without necessitating a complete redesign or significant resource investment Why This Is Important Managing growth. Scalable systems can efficiently handle more users and data without sacrificing speed or reliability. This is particularly important for businesses aiming to expand their customer base.Performance enhancement. By distributing workloads across multiple servers or resources, scalable systems can improve overall performance, leading to faster processing times and better user experiences.Cost-effectiveness. Scalable solutions allow businesses to adjust resources according to demand, helping avoid unnecessary expenditures on infrastructure.Availability assurance. Scalability ensures that systems remain operational even during unexpected spikes in traffic or component failures, which is essential for mission-critical applications.Encouraging innovation. A scalable architecture supports the development of new features and services by minimizing infrastructure constraints. Types of Scalability in General Vertical scaling (scaling up). This involves enhancing the capacity of existing hardware or software components. For example, upgrading a server's CPU or adding more RAM allows it to handle increased loads without changing the overall architecture.Horizontal scaling (scaling out). This method involves adding more machines or instances to distribute the workload. For instance, cloud services allow businesses to quickly add more servers as needed. Challenges Complexity. Designing scalable systems can be complex and may require significant planning and expertise.Cost. Initial investments in scalable technologies can be high, although they often pay off in the long run through improved efficiency.Performance bottlenecks. As systems scale, new bottlenecks may emerge that need addressing, such as database limitations or network congestion. Scalability in Spring Boot Projects Scalability refers to the ability of an application to handle growth — whether in terms of user traffic, data volume, or transaction loads — without compromising performance. In the context of Spring Boot, scalability can be achieved through both vertical scaling (enhancing existing server capabilities) and horizontal scaling (adding more instances of the application). Key Strategies Microservices Architecture Independent services. Break your application into smaller, independent services that can be developed, deployed, and scaled separately. This approach allows for targeted scaling; if one service experiences high demand, it can be scaled independently without affecting others.Spring cloud integration. Utilize Spring Cloud to facilitate microservices development. It provides tools for service discovery, load balancing, and circuit breakers, enhancing resilience and performance under load. Asynchronous Processing Implement asynchronous processing to prevent thread blockage and improve response times. Utilize features like CompletableFuture or message queues (e.g., RabbitMQ) to handle long-running tasks without blocking the main application thread. Asynchronous processing allows tasks to be executed independently of the main program flow. This means that tasks can run concurrently, enabling the system to handle multiple operations simultaneously. Unlike synchronous processing, where tasks are completed one after another, asynchronous processing helps in reducing idle time and improving efficiency. This approach is particularly advantageous for tasks that involve waiting, such as I/O operations or network requests. By not blocking the main execution thread, asynchronous processing ensures that systems remain responsive and performant. Stateless Services Design your services to be stateless, meaning they do not retain any client data between requests. This simplifies scaling since any instance can handle any request without needing session information. There is no stored knowledge of or reference to past transactions. Each transaction is made as if from scratch for the first time. Stateless applications provide one service or function and use a content delivery network (CDN), web, or print servers to process these short-term requests. Database Scalability Database scalability refers to the ability of a database to handle increasing amounts of data, numbers of users, and types of requests without sacrificing performance or availability. A scalable database tackles these database server challenges and adapts to growing demands by either adding resources such as hardware or software, by optimizing its design and configuration, or by undertaking some combined strategy. Type of Databases 1. SQL Databases (Relational Databases) Characteristics: SQL databases are known for robust data integrity and support complex queries.Scalability: They can be scaled both vertically by upgrading hardware and horizontally through partitioning and replication.Examples: PostgreSQL supports advanced features like indexing and partitioning. 2. NoSQL Databases Characteristics: Flexible schema designs allow for handling unstructured or semi-structured data efficiently.Scalability: Designed primarily for horizontal scaling using techniques like sharding.Examples: MongoDB uses sharding to distribute large datasets across multiple servers. Below are some techniques that enhance database scalability: Use indexes. Indexes help speed up queries by creating an index of frequently accessed data. This can significantly improve performance, particularly for large databases. Timescale indexes work just like PostgreSQL indexes, removing much of the guesswork when working with this powerful tool.Partition your data. Partitioning involves dividing a large table into smaller, more manageable parts. This can improve performance by allowing the database to access data more quickly. Read how to optimize and test your data partitions’ size in Timescale.Use buffer cache. In PostgreSQL, buffer caching involves storing frequently accessed data in memory, which can significantly improve performance. This is particularly useful for read-heavy workloads, and while it is always enabled in PostgreSQL, it can be tweaked for optimized performance.Consider data distribution. In distributed databases, data distribution or sharding is an extension of partitioning. It turns the database into smaller, more manageable partitions and then distributes (shards) them across multiple cluster nodes. This can improve scalability by allowing the database to handle more data and traffic. However, sharding also requires more design work up front to work correctly.Use a load balancer. Sharding and load balancing often conflict unless you use additional tooling. Load balancing involves distributing traffic across multiple servers to improve performance and scalability. A load balancer that routes traffic to the appropriate server based on the workload can do this; however, it will only work for read-only queries.Optimize queries. Optimizing queries involves tuning them to improve performance and reduce the load on the database. This can include rewriting queries, creating indexes, and partitioning data. Caching Strategies Caching is vital in enhancing microservices' performance and firmness. It is a technique in which data often and recently used is stored in a separate storage location for quicker retrieval from the main memory, known as a cache. If caching is incorporated correctly into the system architecture, there is a marked improvement in the microservice's performance and a lessened impact on the other systems. When implementing caching: Identify frequently accessed data that doesn't change often — ideal candidates for caching.Use appropriate annotations (@Cacheable, @CachePut, etc.) based on your needs.Choose a suitable cache provider depending on whether you need distributed capabilities (like Hazelcast) or simple local storage (like ConcurrentHashMap).Monitor performance improvements after implementing caches to ensure they're effective without causing additional overheads like stale data issues. Performance Optimization Optimize your code by avoiding blocking operations and minimizing database calls. Techniques such as batching queries or using lazy loading can enhance efficiency. Regularly profile your application using tools like Spring Boot Actuator to identify bottlenecks and optimize performance accordingly. Steps to Identify Bottlenecks Monitoring Performance Metrics Use tools like Spring Boot Actuator combined with Micrometer for collecting detailed application metrics.Integrate with monitoring systems such as Prometheus and Grafana for real-time analysis. Profiling CPU and Memory Usage Utilize profilers like VisualVM, YourKit, or JProfiler to analyze CPU usage, memory leaks, and thread contention.These tools help identify methods that consume excessive resources. Database Optimization Analyze database queries using tools like Hibernate statistics or database monitoring software.Optimize SQL queries by adding indexes, avoiding N+1 query problems, and optimizing connection pool usage. Thread Dump Analysis for Thread Issues Use jstack <pid> command or visual analysis tools like yCrash to debug deadlocks or blocked threads in multi-threaded applications. Distributed Tracing (If Applicable) For microservices architecture, use distributed tracing tools such as Zipkin or Elastic APM to trace latency issues across services. Common Bottleneck Scenarios High Latency Analyze each layer of the application (e.g., controller, service) for inefficiencies. Scenario Tools/Techniques High CPU Usage VisualVM, YourKit High Memory Usage Eclipse MAT, VisualVM Slow Database Queries Hibernate Statistics Network Latency Distributed Tracing Tools Monitoring and Maintenance Continuously monitor your application’s health using tools like Prometheus and Grafana alongside Spring Boot Actuator. Monitoring helps identify performance issues early and ensures that the application remains responsive under load Load Balancing and Autoscaling Use load balancers to distribute incoming traffic evenly across multiple instances of your application. This ensures that no single instance becomes a bottleneck. Implement autoscaling features that adjust the number of active instances based on current demand, allowing your application to scale dynamically Handling 100 TPS in Spring Boot 1. Optimize Thread Pool Configuration Configuring your thread pool correctly is crucial for handling high TPS. You can set the core and maximum pool sizes based on your expected load and system capabilities. Example configuration: Java spring.task.execution.pool.core-size=20 spring.task.execution.pool.max-size=100 spring.task.execution.pool.queue-capacity=200 spring.task.execution.pool.keep-alive=120s This configuration allows for up to 100 concurrent threads with sufficient capacity to handle bursts of incoming requests without overwhelming the system. Each core of a CPU can handle about 200 threads, so you can configure it based on your hardware. 2. Use Asynchronous Processing Implement asynchronous request handling using @Async annotations or Spring WebFlux for non-blocking I/O operations, which can help improve throughput by freeing up threads while waiting for I/O operations to complete. 3. Enable Caching Utilize caching mechanisms (e.g., Redis or EhCache) to store frequently accessed data, reducing the load on your database and improving response times. 4. Optimize Database Access Use connection pooling (e.g., HikariCP) to manage database connections efficiently. Optimize your SQL queries and consider using indexes where appropriate. 5. Load Testing and Monitoring Regularly perform load testing using tools like JMeter or Gatling to simulate traffic and identify bottlenecks. Monitor application performance using Spring Boot Actuator and Micrometer. Choosing the Right Server Choosing the right web server for a Spring Boot application to ensure scalability involves several considerations, including performance, architecture, and specific use cases. Here are key factors and options to guide your decision: 1. Apache Tomcat Type: Servlet containerUse case: Ideal for traditional Spring MVC applications.Strengths: Robust and widely used with extensive community support.Simple configuration and ease of use.Well-suited for applications with a straightforward request-response model.Limitations: May face scalability issues under high loads due to its thread-per-request model, leading to higher memory consumption per request 2. Netty Type: Asynchronous event-driven frameworkUse case: Best for applications that require high concurrency and low latency, especially those using Spring WebFlux.Strengths: Non-blocking I/O allows handling many connections with fewer threads, making it highly scalable.Superior performance in I/O-bound tasks and real-time applications.Limitations: More complex to configure and requires a different programming model compared to traditional servlet-based application. 3. Undertow Type: Lightweight web serverUse case: Suitable for both blocking and non-blocking applications; often used in microservices architectures.Strengths: High performance with low resource consumption.Supports both traditional servlet APIs and reactive programming models.Limitations: Less popular than Tomcat, which may lead to fewer community resources available 4. Nginx (As a Reverse Proxy) Type: Web server and reverse proxyUse case: Often used in front of application servers like Tomcat or Netty for load balancing and serving static content.Strengths: Excellent at handling high loads and serving static files efficiently.Can distribute traffic across multiple instances of your application server, improving scalability Using the Right JVM Configuration 1. Heap Size Configuration The Java heap size determines how much memory is allocated for your application. Adjusting the heap size can help manage large amounts of data and requests. Shell -Xms1g -Xmx2g -Xms: Set the initial heap size (1 GB in this example).-Xmx: Set the maximum heap size (2 GB in this example). 2. Garbage Collection Choosing the right garbage collector can improve performance. The default G1 Garbage Collector is usually a good choice, but you can experiment with others like ZGC or Shenandoah for low-latency requirements. Shell -XX:+UseG1GC For low-latency applications, consider using: Shell -XX:+UseZGC # Z Garbage Collector -XX:+UseShenandoahGC # Shenandoah Garbage Collector 3. Thread Settings Adjusting the number of threads can help handle concurrent requests more efficiently. Set the number of threads in the Spring Boot application properties: Properties files server.tomcat.max-threads=200 Adjust the JVM’s thread stack size if necessary: Shell -Xss512k 4. Enable JIT Compiler Options JIT (Just-In-Time) compilation can optimize the performance of your code during runtime. Shell -XX:+TieredCompilation -XX:CompileThreshold=1000 -XX:CompileThreshold: This option controls how many times a method must be invoked before it's considered for compilation. Adjust according to profiling metrics. Hardware Requirements To support 100 TPS, the underlying hardware infrastructure must be robust. Key hardware considerations include: High-performance servers. Use servers with powerful CPUs (multi-core processors) and ample RAM (64 GB or more) to handle concurrent requests effectively.Fast storage solutions. Implement SSDs for faster read/write operations compared to traditional hard drives. This is crucial for database performance.Network infrastructure. Ensure high bandwidth and low latency networking equipment to facilitate rapid data transfer between clients and servers. Conclusion Performance optimization in Spring Boot applications is not just about tweaking code snippets; it's about creating a robust architecture that scales with growth while maintaining efficiency. By implementing caching, asynchronous processing, and scalability strategies — alongside careful JVM configuration — developers can significantly enhance their application's responsiveness under load. Moreover, leveraging monitoring tools to identify bottlenecks allows for targeted optimizations that ensure the application remains performant as user demand increases. This holistic approach improves user experience and supports business growth by ensuring reliability and cost-effectiveness over time. If you're interested in more detailed articles or references on these topics: Best Practices for Optimizing Spring Boot Application PerformancePerformance Tuning Spring Boot ApplicationsIntroduction to Optimizing Spring HTTP Server Performance

By Mostafa Hosseinzadeh

Automated Bug Fixing: From Templates to AI Agents

If you've spent any time in software development, you know that debugging is often the most time-consuming and frustrating part of the job. What if AI could handle those pesky bugs for you? Recent advances in automated program repair (APR) are making this increasingly realistic. Let's explore how this technology has evolved and where it's headed. The Foundation: Traditional Bug Fixing Approaches Early approaches to automated bug fixing relied on relatively simple principles. Systems like GenProg applied predefined transformation rules to fix common patterns such as null pointer checks or array bounds validation. While innovative for their time, these approaches quickly hit their limits when dealing with complex codebases. Python # Example of a simple template-based fix def fix_array_bounds(code): # Look for array access patterns pattern = r'(\w+)\[(\w+)\]' # Add bounds check replacement = r'(\2 < len(\1) ? \1[\2] : null)' return re.sub(pattern, replacement, code) These early template-based systems faced significant challenges: Limited flexibility. They could only address bugs that matched predefined patterns.Excessive computational cost. Constraint-based methods often ran for hours to produce patches.Poor adaptability. They struggled to handle novel or complex issues in large, dynamic codebases. When Facebook tried implementing template-based repairs for their React codebase, the system struggled with the framework's component lifecycle patterns and state management complexities. Similarly, when used on the Apache Commons library, constraint-based methods often ran for hours to produce patches for even modest-sized functions. The Rise of LLM-Powered Repair The introduction of large language models (LLMs) transformed what's possible in automated bug fixing. Models like GPT-4, Code Llama, DeepSeek Coder, and Qwen2.5 Coder don't just patch syntax errors — they understand the semantic intent of code and generate contextually appropriate fixes across complex codebases. These models bring several capabilities: Context-aware reasoning. They understand relationships between different parts of code.Natural language understanding. They bridge the gap between technical problem statements and actionable fixes.Learning from patterns. They recognize common bug patterns from vast amounts of code. Each model brings unique strengths to the table: LLMKey StrengthIdeal Use CaseGPT-4oAdvanced reasoning and robust code generationEnterprise projects requiring precisionDeepSeekBalance of accuracy and cost-effectivenessSmall-to-medium teams with rapid iterationQwen2.5Strong multilingual support for code repairProjects spanning multiple programming languagesCode LlamaStrong open-source community and customizabilityDiverse programming language environments Three Paradigms of Modern APR Systems 1. Agent-Based Systems Agent-based systems leverage LLMs through multi-agent collaboration, with each agent focusing on a specific role, like fault localization, semantic analysis, or validation. These systems excel at addressing complex debugging challenges through task specialization and enhanced collaboration. The most innovative implementations include: SWE-Agent – Designed for large-scale repository debugging, it can tackle cross-repository dependenciesCODEAGENT – Integrates LLMs with external static analysis tools, optimizing collaborative debugging tasksAgentCoder – An end-to-end modular solution for software engineering tasksSWE-Search – Employs Monte Carlo Tree Search (MCTS) for adaptive path exploration SWE-Search represents a significant advancement with its adaptive path exploration capabilities. It consists of a SWE agent for exploration, a Value Agent for iterative feedback, and a Discriminator Agent for collaborative decision-making. This approach resulted in a 23% relative improvement over standard agents lacking MCTS. 2. Agentless Systems Agentless systems optimize APR by eliminating multi-agent coordination overhead. They operate through a straightforward three-stage process: Hierarchical localization. First, identifying problematic files, then zooming in on classes or functions, and finally, pinpointing specific lines of codeContextual repair. Generating potential patches with appropriate code alterationsValidation. Testing patches using reproduction tests, regression tests, and reranking methods DeepSeek Coder stands out in this category with its repository-level pre-training approach. Unlike earlier methods that operate at the file level, DeepSeek uses repository-level pre-training to better understand cross-file relations and project structures through an innovative dependency parsing algorithm. This model leverages a balanced approach in Fill-in-the-Middle training with a 50% Prefix-Suffix-Middle ratio, boosting both code completion and generation performance. The results speak for themselves — DeepSeek-Coder-Base-33B achieved 50.3% average accuracy on HumanEval and 66.0% on MBPP benchmarks during its initial release. 3. Retrieval-Augmented Systems Retrieval-augmented generation (RAG) systems like CodeRAG blend retrieval mechanisms with LLM-based code generation. These systems incorporate contextual information from GitHub repositories, documentation, and programming forums to support the repair process. Key features include: Contextual retrieval: Pulling relevant information from external knowledge sourcesAdaptive debugging: Supporting repairs involving domain expertise or external API integrationExecution-based validation: Providing functional correctness guarantees through controlled testing environments When evaluated on the SWE benchmark, Agentless systems achieved a 50.8% success rate, outperforming both agent-based approaches (33.6%) and retrieval-augmented methods (30.7%). However, each paradigm has specific strengths depending on the use case and repository complexity. Benchmarking the New Generation Evaluating APR systems requires measuring performance across multiple dimensions: bug-fix accuracy, efficiency, scalability, code quality, and adaptability. Three key benchmarks have emerged: SWE-bench: The All-Round Benchmark SWE-bench tests APR capabilities on real GitHub defects across 12 popular Python repositories. It creates real-world scenarios with problem-solving tasks requiring deep analysis and high accuracy in code edits. Solutions are evaluated using specific test cases in individual repositories for objective rating. CODEAGENTBENCH: Focus on Multi-Agent Frameworks This extension of the SWE-bench targets multi-agent frameworks and repository-level debugging capabilities. It evaluates systems on: Dynamic tool integration – Ability to integrate with static analysis tools and runtimesAgent collaboration – Task specialization and inter-agent communicationExtended scope – Intricate test cases and multi-file challenges CodeRAG-Bench: Testing Retrieval-Augmented Approaches CodeRAG-Bench specifically evaluates systems that integrate contextual retrieval with generation pipelines. It tests adaptability in fixing complex bugs by measuring how well systems incorporate information from diverse sources like GitHub Discussions and documentation. Current Limitations and Challenges Despite impressive advances, APR systems still face significant hurdles: Limited context windows – Processing large codebases (thousands of files) remains challengingAccuracy issues – Multi-line or multi-file edits have higher error rates due to lack of accurate context-sensitive code generationComputational expense – Making large-scale, real-time debugging difficultValidation gaps – Current benchmarks don't fully reflect real-world complexity Real-World Applications The integration of APR into industry workflows has shown significant benefits: Automated version management – Detecting and fixing compatibility issues during upgradesSecurity vulnerability remediation – Pattern recognition and context-aware analysis to speed up patchingTest generation – Creating unit tests for uncovered code paths and integration tests for complex workflows Companies implementing APR tools have reported: 60% reduction in time to fix common problems compared to manual debugging40% increase in test coverage30% reduction in regression bugs Major organizations are taking notice: Google's Gemini Code Assist reports a 40% reduction in time for routine developer tasksMicrosoft's IntelliCode provides context-aware code suggestionsFacebook's SapFix automatically patches bugs in production environments

By Meghana Puvvadi

Building a Weather Service With Genkit and AI

Overview This project demonstrates how to build an AI-enhanced weather service using Genkit, TypeScript, OpenWeatherAPI, and GitHub models. The application showcases modern Node.js patterns and AI integration techniques. Prerequisites Before you begin, ensure you have the following: Node.js installed on your machineGitHub account and access token for GitHub APIsAn OpenWeatherAPI key for fetching weather dataGenkit CLI installed on your machine Technical Deep Dive AI Configuration The core AI setup is initialized with Genkit and GitHub plugin integration. In this case, we are going to use the OpenAI GPT-3 model: TypeScript const ai = genkit({ plugins: [ github({ githubToken: process.env.GITHUB_TOKEN }), ], model: openAIO3Mini, }); Weather Tool Implementation The application defines a custom weather tool using Zod schema validation: TypeScript const getWeather = ai.defineTool( { name: 'getWeather', description: 'Gets the current weather in a given location', inputSchema: weatherToolInputSchema, outputSchema: z.string(), }, async (input) => { const weather = new OpenWeatherAPI({ key: process.env.OPENWEATHER_API_KEY, units: "metric" }) const data = await weather.getCurrent({locationName: input.location}); return `The current weather in ${input.location} is: ${data.weather.temp.cur} Degrees in Celsius`; } ); AI Flow Definition The service exposes an AI flow that processes weather requests: TypeScript const helloFlow = ai.defineFlow( { name: 'helloFlow', inputSchema: z.object({ location: z.string() }), outputSchema: z.string(), }, async (input) => { const response = await ai.generate({ tools: [getWeather], prompt: `What's the weather in ${input.location}?` }); return response.text; } ); Express Server Configuration The application uses the Genkit Express plugin to create an API server: TypeScript const app = express({ flows: [helloFlow], }); Full Code The full code for the weather service is as follows: TypeScript /* eslint-disable @typescript-eslint/no-explicit-any */ import { genkit, z } from 'genkit'; import { startFlowServer } from '@genkit-ai/express'; import { openAIO3Mini, github } from 'genkitx-github'; import {OpenWeatherAPI } from 'openweather-api-node'; import dotenv from 'dotenv'; dotenv.config(); const ai = genkit({ plugins: [ github({ githubToken: process.env.GITHUB_TOKEN }), ], model: openAIO3Mini, }); const weatherToolInputSchema = z.object({ location: z.string().describe('The location to get the current weather for') }); const getWeather = ai.defineTool( { name: 'getWeather', description: 'Gets the current weather in a given location', inputSchema: weatherToolInputSchema, outputSchema: z.string(), }, async (input) => { const weather = new OpenWeatherAPI({ key: process.env.OPENWEATHER_API_KEY, units: "metric" }) const data = await weather.getCurrent({locationName: input.location}); return `The current weather in ${input.location} is: ${data.weather.temp.cur} Degrees in Celsius`; } ); const helloFlow = ai.defineFlow( { name: 'helloFlow', inputSchema: z.object({ location: z.string() }), outputSchema: z.string(), }, async (input) => { const response = await ai.generate({ tools: [getWeather], prompt: `What's the weather in ${input.location}?` }); return response.text; } ); startFlowServer({ flows: [helloFlow] }); Setup and Development 1. Install dependencies: Shell npm install 2. Configure environment variables: Shell GITHUB_TOKEN=your_token OPENWEATHER_API_KEY=your_key 3. Start the development server: Shell npm run genkit:start 4. To run the project in debug mode and set breakpoints, you can run: Shell npm run genkit:start:debug Then, launch the debugger in your IDE. See the .vscode/launch.json file for the configuration. 5. If you want to build the project, you can run: Shell npm run build 6. Run the project in production mode: Shell npm run start:production Dependencies Core Dependencies genkit: ^1.0.5@genkit-ai/express: ^1.0.5openweather-api-node: ^3.1.5genkitx-github: ^1.13.1dotenv: ^16.4.7 Development Dependencies tsx: ^4.19.2typescript: ^5.7.2 Project Configuration Uses ES Modules ("type": "module")TypeScript with NodeNext module resolutionOutput directory: libFull TypeScript support with type definitions License Apache 2.0 Resources Firebase GenkitGitHub ModelsFirebase Express Plugin Conclusion This project demonstrates how to build a weather service using Genkit in Node.js with AI integration. The application showcases modern Node.js patterns and AI integration techniques. You can find the full code of this example in the GitHub repository. Happy coding!

By Xavier Portilla Edo

CORE

How to Build Slack App for Audit Requests

Auditing for any B2B software is crucial for understanding how teams utilize products, measuring efficiency created due to the product usage, security, and data compliance. Organizations may require different types of audits to ensure proper usage, identify potential risks, and support technical engagements. This article will walk you through how to use a technical architecture designed for audits to create a Slack audit application that can facilitate various audit types in a flexible, efficient manner. Overview of the Audit Slack Application The core function of a Slack audit application is to monitor and assess the usage of Slack channels, teams, applications, and integrations. This application can be used to identify specific teams or workspaces, perform detailed audits, and generate reports that help with tasks such as migration or security reviews. The application enables users to interact directly within Slack through a bot interface, making the process seamless and real-time. Types of Audits While sales-focused audits are often part of customer success workflows, the system described can be generalized for any audit purpose. The audit framework can handle tasks such as: Product usage audits. Identifying product usage activity.Compliance audits. Understanding team structures, applications used, and how data flows between different users.Security audits. Monitoring the installation and usage of applications within Slack, assessing their access permissions, and ensuring they are used in compliance with security policies. These audit types are vital for supporting various internal and external processes such as compliance checks, security reviews, or business operations audits. Architecture of the Slack Audit Application The architecture that supports the audit functionality integrates several components to provide seamless access to audit data. Below are the key architectural components: Data Access Layer The audit system interacts with Slack data through secure APIs. It pulls information regarding channels, users, teams, and application usage. In the case of private or tiered data, specific permissions are needed to access restricted information. This data is typically housed in platforms like Salesforce’s Box instance, which provides robust security and large storage capacity for sensitive data. Audit Workflow Users initiate audits via a bot within Slack, reducing the need for external tools. The bot receives input on the audit scope, such as the specific workspace, team ID, or channel to be audited. For more granular control, users can filter the results based on parameters such as whether to include shared external members, channels, or specific user teams. The bot prompts users to input relevant filters like: Team ID – Helps narrow down the scope to specific teams by their raw IDs.Specific channels – Allow users to focus audits on particular Slack channels.Shared members – Option to include external members who belong to Slack Connect channels. After the required data is provided, users can submit their requests, which are processed by the system for analysis. Data Processing The audit requests are processed through a series of workflows that ensure the requested data is accurately retrieved and formatted for reporting. In this context, the architecture must handle large data volumes efficiently, as audits may generate millions of rows of data. To overcome the system’s default row limit (e.g., 400,000 rows), the architecture may need to be enhanced to export larger datasets. In addition, processing sensitive data requires attention to privacy and security. The use of service layers like Uberproxy can help ensure that data is transmitted securely and access is controlled based on the user’s role. Reporting and Access Once the audit is complete, the results are made available to users in the form of reports, which can be accessed via the Slack interface or exported to other tools for further analysis. These reports typically include metadata about channels, team memberships, application usage, and other key insights. Access to the reports can be restricted to authorized users based on roles and permissions. Challenges in Auditing Applications Data privacy and security. Slack audit applications often handle sensitive information, especially when working with external members or private workspaces. To address this, data access should be tightly controlled, ensuring only authorized users can initiate audits or view sensitive data. Solutions like Uberproxy are utilized to enforce security policies and track user activities.Data volume management. Slack’s native limits on the volume of data processed (e.g., 400,000 rows) pose challenges when dealing with large workspaces or high-volume audits. The system may need to be optimized to handle large datasets, either by segmenting the data or using external data stores to temporarily hold and process the results.Audit accuracy. To ensure the accuracy of audits, it is crucial that users input correct values and filters. The system should be designed to validate input before processing, minimizing errors that could delay the workflow or result in incomplete data. Workflow Example Initiate audit. The user interacts with the Slack bot by typing a command like /audit and providing necessary parameters, such as Team ID and the type of audit (channel, team, application).Input filters. The user is prompted to specify additional filters, such as whether to include external members or narrow down the scope to certain channels.Data collection. The system queries the relevant data from Slack and any connected services (e.g., Salesforce Box). The application processes the data to gather details like channel activities, membership data, and integrations.Report generation. After processing, the audit results are compiled into a report and made available through Slack. The report provides actionable insights based on the audit type.Approval and review. Depending on the organization’s needs, the audit report may go through an approval process. The results are reviewed by the relevant stakeholders, and the necessary actions are taken based on the findings. Python async def healthcheck(_req: web.Request): if socket_mode_client is not None and await socket_mode_client.is_connected(): return web.Response(status=200, text="OK") return web.Response(status=503, text="The Socket Mode client is inactive") if __name__ == "__main__": async def start_socket_mode(_web_app: web.Application): handler = AsyncSocketModeHandler(app, secrets.SLACK_APP_TOKEN, ping_interval=60) await handler.connect_async() global socket_mode_client socket_mode_client = handler.client await start_scheduled_task(app.client) async def shutdown_socket_mode(_web_app: web.Application): await socket_mode_client.close() secrets = Secrets() # Initialization app = AsyncApp( token=secrets.SLACK_BOT_TOKEN, signing_secret=secrets.SLACK_SIGNING_SECRET ) register_listeners(app) web_app = app.web_app() web_app.add_routes([web.get("/health", healthcheck)]) web_app.on_startup.append(start_socket_mode) web_app.on_shutdown.append(shutdown_socket_mode) web.run_app(app=web_app, port=80) Conclusion The architecture for building a Slack audit application described above offers a comprehensive, flexible solution for managing various types of audits within Slack. Using Slack’s API, external services, and secure data processing workflows, the system can efficiently track and analyze data across channels, teams, and applications.

By Anuja More

Soaking in the Pressure of Using AI

When large organizations spend billions of dollars in research and development of a revolutionary technology, a time comes when the technology is ready for prime time. The technology giants put their best foot forward to ensure large-scale global adoption of the technology. These are exciting times for any technology enthusiast, and it is natural to feel the urge to be a part of the bandwagon and not feel left out. People across the rank and file of an organization are feeling the pressure of using AI-based solutions for every business problem. How do you soak in the pressure and make the right decisions for your business problems? Let's break down the problem and remove the cobwebs to get a clear picture of when to use AI and when not to. Fundamental Questions to Answer Before you get too deep into the weeds of designing solutions for specific business problems and use cases, there are some very fundamental questions you can ask yourself that will give you very strong hints on whether you need to invest time and energy in building AI solutions. Will investing in an AI-based solution help us disrupt the status quo of our industry?Will it help us leapfrog from an industry laggard to industry leader for any of our products or product features?Can we use AI for any of our product development or product operations processes? Does it significantly reduce the cost of operations in the long run for our product? While this kind of questions are often brought up and answered in board rooms at the executive level, in good companies with a strong product culture, decision-making is democratized, and individual teams are empowered to make key decisions for their product roadmap and product features. Hence, it is important for technical product managers and their teams to think like a CEO and ponder on these fundamental questions when deciding on using AI-based solutions for problems provided by customers and executives. Peeling the Layers Now, let's take the process of questioning and analyzing a layer beneath the fundamental questions and use a framework that will help you decide whether to use AI in your solution design. Predictive AI While there is a lot of content in the public domain on the difference between predictive AI and generative AI, I still want to harp upon the difference between the two. Generative AI (GenAI) has garnered so much attention in the last few months that people often overlook the power of other types of AI, and investment and innovation in those technologies have slowed down in a lot of organizations. It is important for development teams to pause and think about creative solutions for the user/ customer problem at hand. Is the problem you are trying to solve related to gathering historical data of past events, finding patterns in it so that you can predict the outcome of future events? For this kind of problems, predictive AI is where your solutioning river stream should branch out. If you are thinking about Gen AI for such problems, you are barking up the wrong tree. Some common examples of problems that can be solved using predictive AI solutions include financial forecasts, optimum infrastructure utilization, fraud detection, etc. Once you have clarity that the problem you are solving will require a predictive AI-based solution, what kind of machine learning algorithm will solve your problem should be your next logical step in designing the solution. Classification. Assigning data points to predefined classes is a classification problem. For example, flagging any content on a social media platform as inappropriate so that the machine knowledge of appropriate vs inappropriate content is continuously refined is a classification problem.Regression. When the goal is to define the correlation between features and your target variable to predict the future problems is a regression problem. For example, predicting the future traffic on your service to decide when to increase the number of pods.Clustering. Grouping data into different buckets with the goal of defining the right buckets that will help you make decisions is a clustering problem. For example, analyzing customer behavior and use cases to help your organization define the right customer segments is a clustering problem. Isn't it quite amazing that there is so much innovation and creativity you can drive within the realm of predictive AI? So far, in this article, we haven't entered into the realm of generative AI, which has created so much buzz lately. Generative AI In contrast, if the problem you are trying to solve involves training an AI model on your organization's raw data, making it learn by example so that it can take prompts from a user and generate novel outputs, that is where GenAI comes into the picture. The problem might have different permutations and combinations of user prompts and the generated output. The user prompts could be speech, text, or unstructured data, and the generated output could be natural language text, speech, images, or videos. Common examples of problems that can be solved using GenAI include customer support, code generation to reduce product development costs, data synthesis for research and testing, creating on-demand marketing content, etc. When it comes to GenAI-powered solutions, once you are clear that your problem requires a GenAI-based solution, there will be additional important solution design decisions that will help you build the most impactful solution. What LLM base model should I use?What are the latest architectural frameworks and design patterns that i should take advantage of. Think RAG, RLHF, multi-agent react agent using langflow, etc.?How do i ensure the right guardrails of security and data privacy? Conclusion While a lack of technology modernization can make a company's business model obsolete at some point, it is important to ensure that you are using your organization's AI investments for the right problems and at the right places. Forcing AI-based solutions for problems that don't need them can take your product in the wrong direction. Keep your calm and use AI investments wisely.

By Rahul Sharma

Data Engineering

Functions of Data Engineering

AI/ML

Big Data

Data

Databases

IoT

DZone's Featured Data Engineering Resources

The Latest Data Engineering Topics