DZone Spotlight

Monday, December 30 View All Articles »

Transitioning From Postman to Insomnia

By Sharad Jain

As we move away from using Postman, many of us are transitioning to Insomnia for API testing. In this article, I will share how I’ve set up Insomnia to streamline my own workflow. While Insomnia offers a wide range of capabilities beyond HTTP requests, I focus on using it solely for exercising HTTP requests. I hope this provides a useful reference as you configure your own setups. Just to clarify upfront, this isn't an installation guide. Getting Insomnia installed correctly can be a bit tricky, and I, too, have encountered some challenges during the installation and setup process. Several versions of Insomnia are available, and I’ve opted for version 8.1, which I found to be stable. While I’ve also worked with earlier releases, some older versions had notable bugs. If you're encountering issues like being unable to filter or sort your collections or if the "smart" tags aren't functioning as expected, it may help to update to the latest stable version. This guide presents a comprehensive approach to using Insomnia, going beyond the basics to explore advanced techniques. Whether you're a beginner looking to get started or an experienced user seeking to streamline your API workflows, this guide will provide the insights and tools needed to maximize your use of Insomnia. What I Appreciate Most About Insomnia Smooth Startup: Unlike Postman, which often gets stuck with "cannot load" errors, Insomnia launches seamlessly every time.URL Preview and Copy: The ability to preview and easily copy URLs, especially when using variables, is incredibly helpful.Efficient Endpoint Search: Quickly filtering and searching for specific endpoints streamlines the workflow.Pinning Requests: The option to pin requests, though limited to the current folder, helps keep important tasks easily accessible.Flexible Sorting Options: A variety of sorting options in the endpoints panel make it easy to organize and find what I need. Workspaces, Projects, and Collections I use a single workspace — the default "Personal" workspace — but within that workspace, I manage multiple projects. Each project corresponds to a different EAI (Enterprise Application Integration). This setup works well when the hosts or availability zones for each EAI are distinct and unrelated. However, if your EAIs share common hosts or availability zones, consolidating them into a single project may be effective. Otherwise, I recommend creating a separate project for each EAI to maintain clarity and avoid potential conflicts. Each project includes (or will include) a collection tile (for your requests) as well as one or more "documentation" tiles. Testing and other automation processes are outlined within the documentation tile but are beyond the scope of this article. You can return to your projects at any time by clicking the “home” icon. After creating a new project (highlighted in red above), you’ll have the option to create either a new collection or a new document. In most cases, you’ll likely want to choose “collection.” Proxy Configuration First things first: you will likely need to configure the proxy settings. To do this, navigate to the "Preferences" menu (found in the application menu or via the icon in the lower-left corner). Once there, go to the "General" tab, scroll down, and enter the following settings: Now, we’re diving into the core of the matter. For most users, creating and managing requests is the primary function they want to perform in Insomnia. For the remainder of this guide, I’ll be using an example service called “DairyProducts-API.” This service exposes several endpoints, including a variety of “actuator” endpoints, and it’s running across multiple environments in Pivotal Cloud Foundry (PCF) — specifically in the clwdev1 and clwprod1 spaces. These environments include Development, Release, Staging, and Production. To begin working with your requests, locate the "filter" input field in the user interface. To the right of this field, click the + button. This will typically allow you to create either a new HTTP request or a folder to organize your requests. Setting Up Requests for "actuator/info" To monitor and track the Git commit details of each service instance running in different environments (Release, Staging, and Production), we’ll configure a set of requests for the actuator/info endpoint. Steps to Set Up Requests 1. Create a New Request for Each Environment For each environment (Release, Staging, Production), we'll create a separate request. This allows us to check the relevant commit information for each service instance running in those environments. 2. Rename Each Request Rename the request according to the environment to which it corresponds. For example: Release - Actuator InfoStaging - Actuator InfoProduction - Actuator Info 3. Specify the Correct URL for Each Environment You’ll need to define the correct URL for the actuator/info endpoint for each environment. While these are technically “availability zones,” I’m using Insomnia terminology in this guide, where the environment-specific URLs will look something like: https://release.example.com/actuator/infohttps://staging.example.com/actuator/infohttps://production.example.com/actuator/info By setting up these environment-specific requests, you'll be able to easily check the latest Git commit details for each instance across the various environments. Note: This amounts to nine separate requests, each serving the same function but targeting nine different availability zones. And yes, I will be integrating a load balancer later on. For now, however, with the actuator endpoints, the goal is to directly access specific availability zones rather than routing through the GSLB (Global Server Load Balancer) at this stage. The only distinction between each request lies in a segment of the URL. Defining Sub-Environments for Each Availability Zone To define a sub-environment for each availability zone and create a variable to use in the URL, follow these steps: 1. Access the "Manage Environments" Dialog Press 'Control-E' (or click the gear icon, depending on your version) to open the dialog. 2. Create a Sub-Environment for Each Availability Zone There are two types of environments: shared and private. The main difference is that private environments are not exported, but functionally, they are the same as shared ones when not exported. For each sub-environment, create a JSON-formatted document with key-value associations. 3. Define the "az" Variable Add a variable named az for each sub-environment, assigning it the appropriate value for the availability zone. 4. Finalize Your Changes When you're done, click the Close button to save your settings. Pressing "Escape" will discard any unsaved changes. In my case, the value for each az variable is: Release -rel.app.clwdev1.paas.psvc.com Staging -stage.app.clwprod1.paas.psvc.com Production -prod.app.clwprod1.paas.psvc.com Now, I’ll return to clean up my collection by eliminating duplicate entries. Please choose any of the (nearly) duplicate entries to begin. Navigate to the URL and place your cursor at the start of the “AZ” section. Specifically, it should be positioned between products and -rel.Type a slash (just momentarily — this is useful for inserting a variable).Then, type az (or press "CTRL + space") to trigger a pop-up, allowing you to select the appropriate value. Finally, remove the temporary slash you inserted earlier and delete everything from -rel to dairyproducts.com. Your result should look like this: And then this: Click on the 'Query' tab to view the resulting URL after the variable has been expanded. In the upper-left corner, you can select your desired environment, and the URL preview will update accordingly. Next, delete any duplicate requests and rename the remaining request so that the environment name is no longer reflected in the request title. Okta JWT Security Tokens At DairyProducts, we ensure our service endpoints are secured, with some exceptions for health-related endpoints, such as /health, /actuator/info, and /actuator/health. To enforce this security, we use Okta and JSON Web Tokens (JWT), pronounced "jot." Okta is a third-party identity provider that issues (or "mints") tokens that are valid for a specified period, such as two hours. These tokens grant access to the respective services based on the environment in which they were issued. JWT Domains Okta issues JWTs for different environments, and the tokens are restricted by domains: 1. Development/Release (Dev/Rel) Tokens These tokens can be used in both Development and Release environments but cannot be used in Staging or Production. 2. Staging Tokens Tokens issued for Staging are restricted to Staging and cannot be used in Development, Release, or Production. 3. Production Tokens Production tokens are exclusive to the Production environment. This creates three distinct domains from a JWT perspective, ensuring that tokens are scoped appropriately for each environment. Token Requests To issue a JWT for each domain, we’ll need to create three separate requests. Each request requires the following four pieces of information: Issuer URL: The URL where the token is issued by Okta.Client ID: The unique identifier for your application in Okta.Client Secret: The secret key associated with your Client ID, used to authenticate your application.Scope: The scope defines the level of access granted by the token. For example, it might specify permissions related to reading or writing to certain services. Example Folder Structure I have organized my Okta JWT-related configurations and credentials inside a dedicated folder for easy management: Plain Text Okta/JWT/ This folder contains the necessary configurations and secrets required to generate tokens for the different domains. When creating JWT requests, ensure that you have the right issuer URL, client ID, client secret, and scope for the specific environment (Dev/Rel, Staging, or Production). Security Note Make sure that sensitive information, such as your Client ID and Client Secret, is kept secure and never exposed in code repositories or logs. Use environment variables or a secure secrets manager to handle these securely. I’ve put mine inside a separate folder called “Okta/JWT” like this: Now, let’s look at the details for the DevRel request. The “issuer URL” is the URL. Be sure that “/v1/token” is appended at the end. Click on the “Auth” tab and set the authorization type to “basic.” For the “username,” use your “client ID.” For the “password,” use the “client secret.” Click on the “Query” tab and set the following URL arguments: Set “grant_type” to “client_credentials”. Set “response_type” to “token.” Set “scope” as appropriate for your application. Click on the “Headers” tab. Add a header called “Content-Type” with the value “application/x-www-form-urlencoded.” The result looks like this: Now click “Send,” and you should see something like this: Hint: A JWT can contain multiple scope values. By minting a JWT with multiple scopes, you can likely use that same JWT for all services that you intend to call. I’ve created a variable in my “base environment” called “all_scopes,” which contains all of my scopes. So, the URL arguments in my JWT request definition actually look like this: And in my “base environment”: Repeat these steps for the other two requests (for Staging and Production). Secured Request For this next section, I assume you have an endpoint that requires a JWT of your own. Setting this up to access one of my services will not work for you, as you likely have not been authorized to access my services. You need to do this for your own services. Let’s use a JWT in a request to a secured endpoint. First, we’ll do that with a simple solution using copy/paste with the token. Using this method is slow, tedious, and error-prone. Create a new request with your secured URL. Click on the “Auth” tab to open the authorization options. Select “Bearer”. From your “fetch” request, copy the “access_token” value and paste that into the “token” field for the “Bearer” authorization tab. You do not need to provide a prefix. Insomnia uses the default value of “Bearer “ for that prefix. You should now be able to “Send” your request and get a successful response. Using a Variable On our way towards a better solution, let’s put that same token value into a variable. Go into the “Base Environment” dialog and create a variable named jwt_devrel. The value is the same JWT that you used in your endpoint request. Back to the secured endpoint request, replace the token with a variable. Start by clearing out the value that you entered earlier. With the cursor still in the “token” field, start typing jwt (or press control-space). There will be a popup, and you can select the jwt_devrel variable. Once again, you should be able to “Send” your request and get a successful response. Variables and Environment-Specific Contexts This endpoint request is currently “hard-coded” to use the JWT for the Development and Release security domain. Having created JWT requests earlier for each security domain (properly) implies that we will be using the correct JWT (devrel, stg, or prd) for the selected environment. Start by creating two additional variables in the “Base Environment” dialog for jwt_stg and jwt_prd like this: You can leave the additional values empty for now. For the secured request above, we “hard-coded” which JWT to use (jwt_devrel). However, variables can be “chained” (or “nested”). Variable names can be referenced just about anywhere. There are three places to define a variable. In order of precedence, it first checks the folder-specific environment. If not found there, it then checks the current sub-environment (as selected in the environment drop-down). Finally, it checks the “base environment.” If the value contains a (nested) variable, then the process starts over for that portion of the value that contained the variable. (If a variable cannot be resolved, then an error will occur for the null value.) Putting this into action for our secured requests, the bearer token value will be the variable jwt. The sub-environment will define jwt to point to one of jwt_devrel, jwt_stg, or jwt_prd. The resulting jwt_devrel, jwt_stg, or jwt_prd will be defined in the base environment. In a picture, here is what we’re doing: Go back to each of the sub-environments that we created at the beginning and add a new variable called jwt and set the value to one of jwt_devrel (for the Development and Release environments) or jwt_stg (for the staging environments) or jwt_prd (for the production environments). A couple of samples will look like this: Finally, update the request’s authorization value to jwt (allowing the sub-environment selection to properly chain to the desired value). However, you must first select one of the sub-environments in the drop-down, or Insomnia will not be able to resolve the reference. We’re almost there. The request references the variable jwt. The sub-environment resolves jwt to the corresponding security-domain variables, jwt_devrel, jwt_stg or jwt_prd. That last value is defined in the base environment to contain the actual JWT. Automatically Regenerating the JWT The JWT (for jwt_devrel) generated above was valid for about two hours. Hopefully, it’s still current, depending on how quickly you have worked through this guide. But at some point, the JWT will expire. What we’re about to do is to have Insomnia automatically refresh the JWT only when needed. A JWT will not be refreshed when it’s still current but will be refreshed upon expiration. This is the point that we move from banging sticks and rocks together to using the power of Insomnia as a more advanced tool. This is where Insomnia will excel over some other tools (like SoapUI). Let’s review how to get a fresh JWT: When needed.Run the “fetch request that we created at the beginning.”From the response, copy the value from the “access_token” into the value for the jwt_* variable. Here are the steps to automate that process: Open the “Base environment” dialog.Delete the contents of the jwt_devrel variable and position the cursor in the (now) empty quotes.Type control-space to search for a variable.Start typing “response” and wait.From the popup list, select “Response – Body Attribute.” The value will be replaced with an orange placeholder. Double-click on that orange placeholder. The next dialog that appears is the key to automating the JWT refresh. Click on the “Request” field (3rd option) and select the request that corresponds to fetching a dev/rel token: Click on the “Filter” field and type: “$.access_token” (exactly). Set the refresh trigger to be when the existing token expires. Set the TTL (time-to-live) or allowable age for a token. This is set manually in this input. For two hours, the value is 7200 seconds. (Optional) Click on the “refresh” icon to test the refresh. The result should be what appears to be a JWT. And you’re done (for the Dev/Release tokens)! Repeat this same process but for the staging and production equivalents. These are some of the important areas of Insomnia that I have explored and found to be useful. Conclusion To wrap things up, while Postman has been the first and most popular choice for API testing for a long time, Insomnia is gaining momentum to become a favorite for many developers, and for a good reason. Its simple, intuitive interface and powerful features — like easy environment management, built-in debugging, and support for plugins — make it a great choice for anyone looking to work more efficiently. Insomnia isn’t just an alternative to Postman; it often offers a smoother, faster experience, especially if you’re looking for something lightweight and flexible. Whether you’re dealing with REST, GraphQL, or gRPC, Insomnia has the tools to keep you productive. If you haven’t tried it yet, it’s worth checking out — your development workflow might just thank you for it. More

Building a Tic Tac Toe Game Using React

By Maulik Suchak

Welcome to my version of the classic Tic Tac Toe game! I created a Tic Tac Toe game using React and SCSS, completing it in around 6 hours. The game features single-player mode against the computer, a winner announcement popup, and win counters for both players. While the grid is customizable to n x n, the current winning logic supports only 3 x 3. Future improvements include smarter computer moves and real-time multiplayer functionality. Overview Total development time needed: ~6 hoursTechnologies used: JavaScript (React) + SCSS Game Rules 1. Objective The goal of the game is to align three of your symbols (X or O) in a horizontal, vertical, or diagonal row on a 3 x 3 grid. 2. Turn Order Player A always starts and plays as X.The computer opponent plays as O and makes its move immediately after Player A. 3. Gameplay Players take turns marking empty cells on the grid.The first player to align three symbols wins the game. 4. Winning Conditions A player wins if they achieve: Three symbols in a horizontal rowThree symbols in a vertical column, orThree symbols in a diagonal line 5. Draw If all cells are filled and no player meets the winning conditions, the game ends in a draw. 6. Score Tracking Win counters keep track of the total number of victories for both Player A and the computer. Features Implemented (MVP) Here’s what’s included in this version of the game: Play against the computer: A fun single-player mode where you take on the computerWinner announcement popup: At the end of each game, a popup reveals the winner.Win counters: Track total wins for both the player and the computer.Customizable grid: Create an n x n grid (though the winning logic currently supports only 3 x 3). Possible Enhancements Looking ahead, here are a few areas for improvement: [UX] Local storage for scores: Save game scores locally so they persist after refreshing the browser.[UX/Tech] Smarter computer moves: Replace random moves with an algorithm for more competitive gameplay.[UX] Real-time multiplayer: Enable two players to compete in real-time using technologies like Socket.io.[Tech] Advanced SCSS organization: Refactor SCSS using mixins, variables, and better structuring techniques.[UX/Tech] Graceful degradation: Ensure compatibility with non-modern browsers for broader support. Setup and Development Prerequisites Before diving in, ensure you have Node.js installed (recommended version: 6.4.1). Getting Started Clone the repository from GitHub. 2. Install dependencies: npm install 3. Start the development server: npm start Open http://localhost:3000 in your browser to view the app. 4. (Optional) Build the project for production: npm run build This creates an optimized build folder, ready for deployment. Testing Run tests in interactive watch mode with: npm test Technical Details File Structure The project follows the standard Create React App structure with a modular component-based approach. src ├── index.js ├── App.css ├── App.js ├── Components │ └── Board │ ├── Board.scss │ ├── Cross.jsx │ ├── ScoreBoard.jsx │ ├── index.jsx │ ├── Round.jsx │ └── Footer │ ├── Footer.scss │ ├── index.jsx │ └── Header │ ├── Header.scss │ ├── index.jsx ├── utils │ └── lib.js public ├── index.html Description and Approach The app was bootstrapped with Create React App for a quick MVP setup.The game initializes an n x n grid for gameplay.Each cell has a click event listener, allowing Player A (X) to make the first move.The computer’s turn is simulated with a random cell selection logic, which will need enhancement for custom grid sizes.Used SCSS for styling with a focus on speed over complexity during development. Testing Approach Manual Tests Ensure the computer makes a move after Player A finishes.Verify the game ends with a winner announcement banner.Test automated moves by the computer. Potential Unit Tests Check if the computer moves after Player A.Validate that the game displays the correct winner.Test edge cases for grid initialization and user interactions. Conclusion Thank you for exploring my project! You can find the complete source code and contribute to this project on GitHub. Let’s build cool stuff together! More

Trend Report

Observability and Performance

The dawn of observability across the software ecosystem has fully disrupted standard performance monitoring and management. Enhancing these approaches with sophisticated, data-driven, and automated insights allows your organization to better identify anomalies and incidents across applications and wider systems. While monitoring and standard performance practices are still necessary, they now serve to complement organizations' comprehensive observability strategies. This year's Observability and Performance Trend Report moves beyond metrics, logs, and traces — we dive into essential topics around full-stack observability, like security considerations, AIOps, the future of hybrid and cloud-native observability, and much more.

Refcard #400

Java Application Containerization and Deployment

By Mark Heckler

Refcard #392

Software Supply Chain Security

By Justin Albano

CORE

AWS Performance Tuning: Why EC2 Autoscaling Isn’t a Silver Bullet

AWS EC2 Autoscaling is frequently regarded as the ideal solution for managing fluctuating workloads. It offers automatic adjustments of computing resources in response to demand, theoretically removing the necessity for manual involvement. Nevertheless, depending exclusively on EC2 Autoscaling can result in inefficiencies, overspending, and performance issues. Although Autoscaling is an effective tool, it does not serve as a one-size-fits-all remedy. Here’s a comprehensive exploration of why Autoscaling isn’t a guaranteed fix and suggestions for engineers to improve its performance and cost-effectiveness. The Allure of EC2 Autoscaling Autoscaling groups (ASGs) dynamically modify the number of EC2 instances to align with your application’s workload. This feature is ideal for unpredictable traffic scenarios, like a retail site during a Black Friday rush or a media service broadcasting a live event. The advantages are evident: Dynamic scaling: Instantly adds or removes instances according to policies or demand.Cost management: Shields against over-provisioning in low-traffic times.High availability: Guarantees that applications stay responsive during peak load. Nonetheless, these benefits come with certain limitations. The Pitfalls of Blind Reliance on Autoscaling 1. Cold Start Delays Autoscaling relies on spinning up new EC2 instances when demand increases. This process involves: Booting up a virtual machine.Installing or configuring necessary software.Connecting the instance to the application ecosystem. In many cases, this can take several minutes — an eternity during traffic spikes. For example: An e-commerce platform experiencing a flash sale might see lost sales and frustrated customers while waiting for new instances to come online.A real-time analytics system could drop critical data points due to insufficient compute power during a sudden surge. Solution: Pre-warm instances during expected peaks or use predictive scaling based on historical patterns. 2. Inadequate Load Balancing Even with Autoscaling in place, improperly configured load balancers can lead to uneven traffic distribution. For instance: A health-check misconfiguration might repeatedly route traffic to instances that are already overloaded.Sticky sessions can lock users to specific instances, negating the benefits of new resources added by Autoscaling. Solution: Pair Autoscaling with robust load balancer configurations, such as application-based routing and failover mechanisms. 3. Reactive Nature of Autoscaling Autoscaling policies are inherently reactive — they respond to metrics such as CPU utilization, memory usage, or request counts. By the time the system recognizes the need for additional instances, the spike has already impacted performance. Example: A fintech app processing high-frequency transactions saw delays when new instances took 5 minutes to provision. This lag led to compliance violations during market surges. Solution: Implement predictive scaling using AWS Auto Scaling Plans or leverage AWS Lambda for instantaneous scaling needs where possible. 4. Costs Can Spiral Out of Control Autoscaling can inadvertently cause significant cost overruns: Aggressive scaling policies may provision more resources than necessary, especially during transient spikes.Overlooked instance termination policies might leave idle resources running longer than intended. Example: A SaaS platform experienced a 300% increase in cloud costs due to Autoscaling misconfigurations during a product launch. Instances remained active long after the peak traffic subsided. Solution: Use AWS Cost Explorer to monitor spending and configure instance termination policies carefully. Consider Reserved or Spot Instances for predictable workloads. Enhancing Autoscaling for Real-World Efficiency To overcome these challenges, Autoscaling must be part of a broader strategy: 1. Leverage Spot and Reserved Instances Use a mix of Spot, Reserved, and On-Demand Instances. For example, Reserved Instances can handle baseline traffic, while Spot Instances handle bursts, reducing costs. 2. Combine With Serverless Architectures Serverless services like AWS Lambda can absorb sudden, unpredictable traffic bursts without the delay of provisioning EC2 instances. For instance, a news website might use Lambda to serve spikes in article views after breaking news. 3. Implement Predictive Scaling AWS’s predictive scaling uses machine learning to forecast traffic patterns. A travel booking site, for example, could pre-scale instances before the surge in bookings during holiday seasons. 4. Optimize Application Performance Sometimes the root cause of scaling inefficiencies lies in the application itself: Inefficient code.Database bottlenecks.Overuse of I/O operations.Invest in application profiling tools like Amazon CloudWatch and AWS X-Ray to identify and resolve these issues. The Verdict EC2 Autoscaling is an essential component of modern cloud infrastructure, but it’s not a perfect solution. Cold start delays, reactive scaling, and cost overruns underscore the need for a more holistic approach to performance tuning. By combining Autoscaling with predictive strategies, serverless architectures, and rigorous application optimization, organizations can achieve the scalability and cost-efficiency they seek. Autoscaling is an impressive tool, but like any tool, it’s most effective when wielded thoughtfully. For engineers, the challenge is not whether to use Autoscaling but how to use it in harmony with the rest of the AWS ecosystem.

By John Akkarakaran Jose

Point-In-Time Recovery (PITR) in PostgreSQL

Point-in-time recovery (PITR) is a robust feature in PostgreSQL that has become even more efficient and user-friendly with the advent of PostgreSQL. It enables administrators to restore a PostgreSQL database to a specific moment in the past. This is particularly useful if you manage disaster recovery for a large-scale system with a large transaction load. This blog will explore PITR and equip you with knowledge about potential pitfalls and their solutions, ensuring a smooth and successful implementation. We'll also share its key benefits and detail a step-by-step implementation of PostgreSQL. Key Components Implementing PITR involves two key components: 1. Base Backup A base backup is a snapshot of the database at a specific point in time. It includes all the data files, configuration files, and metadata required to restore the database to its original state. The base backup serves as the starting point for PITR. 2. Write-Ahead Logs (WAL) WAL files record every change made to the database. These logs store the changes required to recover the database to its state at a specific time. When you perform a PITR, you replay the WAL files sequentially to recreate the desired database state. Why Use PITR? PITR is beneficial in several scenarios: Undo Accidental Changes Accidental operations, such as a DELETE or DROP statement without a WHERE clause, can result in significant data loss. With PITR, you can recover the database to a state just before the mistake, preserving critical data. Recover From Data Corruption Application bugs, hardware failures, or disk corruption can cause data inconsistencies. PITR allows you to restore a clean database snapshot and replay only valid changes, minimizing downtime and data loss. Restore for Testing or Debugging Developers often need to replicate a production database for debugging or testing purposes. PITR enables the creation of a snapshot of the database at a specific point, facilitating controlled experiments without affecting live data. Disaster Recovery PITR is essential for disaster recovery strategies. In catastrophic failures, such as natural disasters or cyberattacks, you can quickly restore the database to its last consistent state, ensuring business continuity. Efficient Use of Resources By combining periodic base backups with WAL files, PITR minimizes the need for frequent full backups, saving storage space and reducing backup times. PITR is also an exact recovery method, allowing you to recover to a specific second and minimizing the risk of data loss during an incident. It is flexible enough to handle diverse recovery scenarios, from a single transaction rollback to a full database restore efficiently. What’s New in PostgreSQL 17 for PITR? PostgreSQL 17 introduces several enhancements for PITR, focusing on performance, usability, and compatibility: Failover Slot Synchronization Logical replication slots now support synchronization during failovers. This ensures that WALs required for PITR are retained even after a failover, reducing manual intervention. Enhanced WAL Compression The WAL compression algorithm has been updated to improve storage efficiency, reducing the space required for archiving WALs. This is particularly beneficial for large-scale systems with high transaction rates. Faster Recovery Speeds Optimizations in the WAL replay process result in faster recovery times, particularly for large data sets. Improved Compatibility With Logical Replication PITR now integrates better with logical replication setups, making it easier to recover clusters that leverage physical and logical replication. Granular WAL Archiving Control PostgreSQL 17 offers more control over WAL archiving, allowing you to fine-tune the retention policies to match recovery requirements. Detailed Steps to Perform PITR in PostgreSQL Follow these steps to set up and perform PITR. Before using PITR, you'll need: WAL archiving: Enable and configure WAL archiving.Base backup: Take a complete base backup using pg_basebackup or pgBackRest.Secure storage: Ensure backups and WAL files are stored securely, preferably off-site. 1. Configure WAL Archiving WAL archiving is critical for PITR as it stores the incremental changes between backups. To configure WAL archiving, update the postgresql.conf file, setting: Shell wal_level = replica # Ensures sufficient logging for recovery archive_mode = on # Enables WAL archiving archive_command = 'cp %p /path/to/wal_archive/%f' # Command to archive WALs max_wal_senders = 3 # Allows replication and archiving Then, after setting the configuration parameters, restart the PostgreSQL server: Shell sudo systemctl restart postgresql Check the status of WAL archiving with the following command: SQL SELECT * FROM pg_stat_archiver; Look for any errors in the pg_stat_archiver view or PostgreSQL logs. 2. Perform a Base Backup Take a base backup to use as the starting point for PITR; using pg_basebackup, the command takes the form: Shell pg_basebackup -D /path/to/backup_directory -Fp -Xs -P This creates a consistent database snapshot and ensures that WAL files are archived for recovery. 3. Validate the Backup Integrity Use pg_verifybackup to validate the integrity of your backup: Shell pg_verifybackup /path/to/backup_directory 4. Simulate a Failure For demonstration purposes, you can simulate a failure. For example, accidentally delete data: Shell DELETE FROM critical_table WHERE id = 123; 5. Restore the Base Backup Before restoring the base backup, stop the PostgreSQL server: Shell sudo systemctl stop postgresql Then, use the following command to change the name of the existing data directory: Shell mv /var/lib/pgsql/17/data /var/lib/pgsql/17/data_old Then, replace the data directory with the base backup: Shell cp -r /path/to/backup_directory /var/lib/pgsql/17/data Update the permissions on the data directory: Shell chown -R postgres:postgres /var/lib/pgsql/17/data 6. Configure Recovery To enable recovery mode, you first need to create a recovery.signal file in the PostgreSQL data directory: Shell touch /var/lib/pgsql/17/data/recovery.signal Then, update postgresql.conf, adding the following parameters: Shell restore_command = 'cp /path/to/wal_archive/%f "%p"' # Restore archived WALs recovery_target_time = '2024-11-19 12:00:00' # Specify target time Alternatively, use recovery_target_lsn or recovery_target_name for more advanced scenarios. 7. Start PostgreSQL in Recovery Mode Restart the PostgreSQL server with the command: Shell sudo systemctl start postgresql Monitor the logs for recovery progress: Shell tail -f /var/lib/pgsql/17/pg_log/postgresql.log PostgreSQL will automatically exit recovery mode and become operational when recovery is complete. 8. Verify Recovery After recovery, validate the database state: SQL SELECT * FROM critical_table WHERE id = 123; Addressing Potential Issues Missing or Corrupted WAL Files Problem WAL files required for recovery are missing or corrupted. Solution Ensure backups and WAL archives are validated regularly using tools like pg_verifybackup.Use redundant storage for WAL archives. Incorrect Recovery Target Problem Recovery stops at an unintended state. Solution Double-check the recovery_target_time, recovery_target_lsn, or recovery_target_name.Use pg_waldump to inspect WAL files for target events. Performance Bottlenecks During Recovery Problem Recovery takes too long due to large WAL files. Solution Optimize recovery performance by increasing maintenance_work_mem and max_parallel_workers.Use WAL compression to reduce file size. Clock Skew Issues Problem Recovery timestamps need to be aligned due to clock differences. Solution Synchronize server clocks using tools like NTP. Misconfigured WAL Archiving Problem Improper archive_command causes WAL archiving failures. Solution Test the archive_command manually: cp /path/to/test_wal /path/to/wal_archive/.Ensure sufficient permissions for the archive directory. Best Practices for PITR Automate backups: Use tools like pgBackRest or Barman for scheduled backups and WAL archiving.Monitor WAL archiving: Regularly check pg_stat_archiver for issues.Validate backups: Always verify backup integrity using pg_verifybackup.Test recovery procedures: Regularly simulate recovery scenarios to ensure readiness.Secure WAL archives: For WAL archives, use secure, redundant storage, such as cloud services or RAID-configured disks. Conclusion Point-in-time recovery (PITR) is critical for maintaining database reliability and mitigating data loss in the event of an incident. pgEdge and PostgreSQL 17’s enhancements make PITR faster, more efficient, and easier to manage, particularly for large-scale or highly available systems. Following this guide's steps and best practices will help you implement and manage PITR effectively in your PostgreSQL environments. Regular testing and monitoring are essential to ensure that recovery processes are available when you need them most.

By Ibrar Ahmed

Challenges and Solutions in Developing Real-Time Messaging Systems

Social media networks, service marketplaces, and online shops all rely heavily on real-time messaging. Instant communication is essential for platforms like TaskRabbit, Thumbtack, and a multitude of others. Real-time interactions accelerate growth and foster user engagement, making messaging features pivotal for any business to succeed online. Yet, building a real-time messaging system is anything but simple. The intricate complexities of creating and successfully integrating said feature imply plenty of underwater rocks, and modern developers are forced to look for effective and highly scalable solutions to this conundrum. This article focuses on some of the common issues developers encounter and offers tangible solutions to those problems. Understanding the Complexity of Real-Time Messaging Systems When talking about real-time messaging, it’s important to remember that texting is but one of the cornerstones behind the overall architecture. The key to success is maintaining a reliable and continuous flow of data while keeping responsiveness and speed at a level that adheres to every user’s expectations. Even the slightest delay can have a detrimental impact on the user experience, particularly in critical scenarios. Understanding those intricacies is crucial, and that’s why we’ll review those challenges one by one, providing effective solutions that could potentially solve most of the issues effectively. Challenge 1: Scalability for Growing User Bases Any platform with high user engagement accumulates a larger audience over time. The user base grows as more people join, and the built-in messaging feature has to process larger volumes of data, preferably without affecting its overall performance. Even the slightest mishap could lead to severe latency issues, with dropped messages, leading to a poor user experience. Solution: Microservices and Load Distribution Tackling scalability implies adopting a microservices architecture. That way, every single component of the messaging system will operate independently. This results in greater scalability potential when services adapt to changing traffic without any drops in performance. Traffic distribution can be managed with the help of load balancers so that no single server would become overwhelmed. Furthermore, managing a high throughput of messages can be done with message brokers like Apache Kafka or RabbitMQ. They ensure a smooth data flow even under heavier loads. Challenge 2: Achieving Low Latency Latency must be kept to a minimum to ensure a truly seamless and uninterrupted messaging experience for the users. People want their messages to be delivered instantly, so higher latency could spoil the experience and lead to users’ dissatisfaction. Solution: WebSockets and Efficient Protocols Replacing traditional HTTP with WebSockets could remedy the issue and reduce latency to a minimum. WebSockets enable real-time, two-way communication by maintaining an open connection between the client and server. That, minus the overhead of constantly opening and closing new connections. Using Protocol Buffers instead of JSON for data transfer can accelerate message transmission. Developers can reduce the size of each message payload, ensuring more efficient and much faster data transfer. Challenge 3: Ensuring High Availability and Reliability Any downtime could be critical for real-time user communication. Hence, the system must be ready to function continuously, even if one or several components malfunction or experience a heavier traffic load. Solution: Distributed and Redundant Systems High availability can be ensured with the help of a distributed architecture. Databases like Apache Cassandra or Amazon DynamoDB can provide the required redundancy, keeping the system operational even in case of server issues. In addition, a graceful degradation strategy coupled with circuit breakers could prevent a wave of small issues from turning into one huge problem that could potentially lead to a system-wide failure. That way, if one service fails, the system itself will not crash altogether. Challenge 4: Managing Security and User Privacy Cybersecurity is pivotal. It’s particularly important when dealing with private communication. Yet, real-time messaging systems are susceptible to a variety of security issues. That includes data interception, unauthorized access, or even Denial of Service (DoS) attacks. Solution: End-to-End Encryption and Secure Authentication Protecting user data could be achieved by implementing end-to-end encryption. That way, only the intended recipients will be able to view messages. Furthermore, secure authentication mechanisms like OAuth 2.0 or JWT (JSON Web Tokens) will prevent unauthorized access and protect user accounts effectively. Also, detecting and effectively mitigating potential DoS attacks can be achieved through rate limiting and monitoring tools. They ensure that the system remains fully operational and secure from threats. Challenge 5: Handling Offline Users and Synchronization There’s also the issue of users going offline and back online again. The messaging system, therefore, needs to act accordingly and ensure that every message is delivered correctly. That is, even if there are network interruptions. Solution: Message Queuing and Asynchronous Storage Message queues like Amazon SQS or Apache Kafka ensure that messages are stored and delivered reliably, regardless of whether the recipient is online. Furthermore, databases like MongoDB could handle asynchronous storage and retrieval. It would sync messages effectively when the user returns online. Building a Robust Real-Time Messaging System Real-time messaging systems aren’t just about user interface or coding. They are complex systems that require a thought-out approach, and scalability, security, and robust architecture are all part of the package. Any online marketplace or social platform that requires real-time messaging must deliver a seamless and truly secure user experience. That way, users will be kept satisfied and engaged, accelerating platform growth and helping its creators succeed in every way possible.

By Daniela Aroundwire

How OpenAI’s Downtime Incident Teaches Us to Build More Resilient Systems

On December 11, 2024, OpenAI services experienced significant downtime due to an issue stemming from a new telemetry service deployment. This incident impacted API, ChatGPT, and Sora services, resulting in service disruptions that lasted for several hours. As a company that aims to provide accurate and efficient AI solutions, OpenAI has shared a detailed post-mortem report to transparently discuss what went wrong and how they plan to prevent similar occurrences in the future. In this article, I will describe the technical aspects of the incident, break down the root causes, and explore key lessons that developers and organizations managing distributed systems can take away from this event. The Incident Timeline Here’s a snapshot of how the events unfolded on December 11, 2024: Time (PST)Event3:16 PMMinor customer impact began; service degradation observed3:27 PMEngineers began redirecting traffic from impacted clusters3:40 PMMaximum customer impact recorded; major outages across all services4:36 PMFirst Kubernetes cluster began recovering5:36 PMSubstantial recovery for API services began5:45 PMSubstantial recovery for ChatGPT observed7:38 PMAll services fully recovered across all clusters Figure 1: OpenAI Incident Timeline - Service Degradation to Full Recovery. Root Cause Analysis The root of the incident lay in a new telemetry service deployed at 3:12 PM PST to improve the observability of Kubernetes control planes. This service inadvertently overwhelmed Kubernetes API servers across multiple clusters, leading to cascading failures. Breaking It Down Telemetry Service Deployment The telemetry service was designed to collect detailed Kubernetes control plane metrics, but its configuration unintentionally triggered resource-intensive Kubernetes API operations across thousands of nodes simultaneously. Overloaded Control Plane The Kubernetes control plane, responsible for cluster administration, became overwhelmed. While the data plane (handling user requests) remained partially functional, it depended on the control plane for DNS resolution. As cached DNS records expired, services relying on real-time DNS resolution began failing. Insufficient Testing The deployment was tested in a staging environment, but the staging clusters did not mirror the scale of production clusters. As a result, the API server load issue went undetected during testing. How the Issue Was Mitigated When the incident began, OpenAI engineers quickly identified the root cause but faced challenges implementing a fix because the overloaded Kubernetes control plane prevented access to the API servers. A multi-pronged approach was adopted: Scaling Down Cluster Size: Reducing the number of nodes in each cluster lowered the API server load.Blocking Network Access to Kubernetes Admin APIs: Prevented additional API requests, allowing servers to recover.Scaling Up Kubernetes API Servers: Provisioning additional resources helped clear pending requests. These measures enabled engineers to regain access to the control planes and remove the problematic telemetry service, restoring service functionality. Lessons Learned This incident highlights the criticality of robust testing, monitoring, and fail-safe mechanisms in distributed systems. Here’s what OpenAI learned (and implemented) from the outage: 1. Robust Phased Rollouts All infrastructure changes will now follow phased rollouts with continuous monitoring. This ensures issues are detected early and mitigated before scaling to the entire fleet. 2. Fault Injection Testing By simulating failures (e.g., disabling the control plane or rolling out bad changes), OpenAI will verify that their systems can recover automatically and detect issues before impacting customers. 3. Emergency Control Plane Access A “break-glass” mechanism will ensure engineers can access Kubernetes API servers even under heavy load. 4. Decoupling Control and Data Planes To reduce dependencies, OpenAI will decouple the Kubernetes data plane (handling workloads) from the control plane (responsible for orchestration), ensuring that critical services can continue running even during control plane outages. 5. Faster Recovery Mechanisms New caching and rate-limiting strategies will improve cluster startup times, ensuring quicker recovery during failures. Sample Code: Phased Rollout Example Here’s an example of implementing a phased rollout for Kubernetes using Helm and Prometheus for observability. Helm deployment with phased rollouts: Shell # Deploy the telemetry service to 10% of clusters helm upgrade --install telemetry-service ./telemetry-chart \ --set replicaCount=10 \ --set deploymentStrategy=phased-rollout Prometheus query for monitoring API server load: YAML # PromQL Query to monitor Kubernetes API server load sum(rate(apiserver_request_duration_seconds_sum[1m])) by (cluster) / sum(rate(apiserver_request_duration_seconds_count[1m])) by (cluster) This query helps track response times for API server requests, ensuring early detection of load spikes. Fault Injection Example Using chaos-mesh, OpenAI could simulate outages in the Kubernetes control plane. Shell # Inject fault into Kubernetes API server to simulate downtime kubectl create -f api-server-fault.yaml api-server-fault.yaml: YAML apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: api-server-fault spec: action: pod-kill mode: one selector: namespaces: - kube-system labelSelectors: app: kube-apiserver This configuration intentionally kills an API server pod to verify system resilience. What This Means for You This incident underscores the importance of designing resilient systems and adopting rigorous testing methodologies. Whether you manage distributed systems at scale or are implementing Kubernetes for your workloads, here are some takeaways: Simulate Failures Regularly: Use chaos engineering tools like Chaos Mesh to test system robustness under real-world conditions.Monitor at Multiple Levels: Ensure your observability stack tracks both service-level metrics and cluster health metrics.Decouple Critical Dependencies: Reduce reliance on single points of failure, such as DNS-based service discovery. Conclusion While no system is immune to failures, incidents like this remind us of the value of transparency, swift remediation, and continuous learning. OpenAI’s proactive approach to sharing this post-mortem provides a blueprint for other organizations to improve their operational practices and reliability. By prioritizing robust phased rollouts, fault injection testing, and resilient system design, OpenAI is setting a strong example of how to handle and learn from large-scale outages. For teams that manage distributed systems, this incident is a great case study of how to approach risk management and minimize downtime for core business processes. Let’s use this as an opportunity to build better, more resilient systems together.

By Vasanthi Govindaraj

Creating Scalable, Compliant Cloud Data Pipelines in SaaS through AI Integration

Data management is undergoing a rapid transformation and is emerging as a critical factor in distinguishing success within the Software as a Service (SaaS) industry. With the rise of AI, SaaS leaders are increasingly turning to AI-driven solutions to optimize data pipelines, improve operational efficiency, and maintain a competitive edge. However, effectively integrating AI into data systems goes beyond simply adopting the latest technologies. It requires a comprehensive strategy that tackles technical challenges, manages complex real-time data flows, and ensures compliance with regulatory standards. This article will explore the journey of building a successful AI-powered data pipeline for a SaaS product. We will cover everything from initial conception to full-scale adoption, highlighting the key challenges, best practices, and real-world use cases that can guide SaaS leaders through this critical process. 1. The Beginning: Conceptualizing the Data Pipeline Identifying Core Needs The first step in adopting AI-powered data pipelines is understanding the core data needs of your SaaS product. This involves identifying the types of data the product will handle, the specific workflows involved, and the problems the product aims to solve. Whether offering predictive analytics, personalized recommendations, or automating operational tasks, each use case will influence the design of the data pipeline and the AI tools required for optimal performance. Data Locality and Compliance Navigating the complexities of data locality and regulatory compliance is one of the initial hurdles for SaaS companies implementing AI-driven data pipelines. Laws such as the GDPR in Europe impose strict guidelines on how companies handle, store, and transfer data. SaaS leaders must ensure that both the storage and processing locations of data comply with regulatory standards to avoid legal and operational risks. Data Classification and Security Managing data privacy and security involves classifying data based on sensitivity (e.g., personally identifiable information or PII vs. non-PII) and applying appropriate access controls and encryption. Here are some essential practices for compliance: Key Elements of a Robust Data Protection Strategy By addressing these challenges, SaaS companies can build AI-driven data pipelines that are secure, compliant, and resilient. 2. The Build: Integrating AI into Data Pipelines Leveraging Cloud for Scalable and Cost-Effective AI-Powered Data Pipelines To build scalable, efficient, and cost-effective AI-powered data pipelines, many SaaS companies turn to the cloud. Cloud platforms offer a wide range of tools and services that enable businesses to integrate AI into their data pipelines without the complexity of managing on-premises infrastructure. By leveraging cloud infrastructure, companies gain flexibility, scalability, and the ability to innovate rapidly, all while minimizing operational overhead and avoiding vendor lock-in. Key Technologies in Cloud-Powered AI Pipelines An AI-powered data pipeline in the cloud typically follows a series of core stages, each supported by a set of cloud services: End-to-End Cloud Data Pipeline 1. Data Ingestion The first step in the pipeline is collecting raw data from various sources. Cloud services allow businesses to easily ingest data in real time from internal systems, customer interactions, IoT devices, and third-party APIs. These services can handle both structured and unstructured data, ensuring that no valuable data is left behind. 2. Data Storage Once data is ingested, it needs to be stored in an optimized manner for processing and analysis. Cloud platforms provide flexible storage options, such as: Data Lakes: For storing large volumes of raw, unstructured data that can later be analyzed or processed.Data Warehouses: For storing structured data, performing complex queries, and reporting.Scalable Databases: For storing key-value or document data that needs fast and efficient access. 3. Data Processing After data is stored, it needs to be processed. The cloud offers both batch and real-time data processing capabilities: Batch Processing: For historical data analysis, generating reports, and performing large-scale computations.Stream Processing: For real-time data processing, enabling quick decision-making and time-sensitive applications, such as customer support or marketing automation. 4. Data Consumption The final stage of the data pipeline is delivering processed data to end users or business applications. Cloud platforms offer various ways to consume the data, including: Business Intelligence Tools: For creating dashboards, reports, and visualizations that help business users make informed decisions.Self-Service Analytics: Enabling teams to explore and analyze data independently.AI-Powered Services: Delivering real-time insights, recommendations, and predictions to users or applications. Ensuring a Seamless Data Flow A well-designed cloud-based data pipeline ensures smooth data flow from ingestion through to storage, processing, and final consumption. By leveraging cloud infrastructure, SaaS companies can scale their data pipelines as needed, ensuring they can handle increasing volumes of data while delivering real-time AI-driven insights and improving customer experiences. Cloud platforms provide a unified environment for all aspects of the data pipeline — ingestion, storage, processing, machine learning, and consumption — allowing SaaS companies to focus on innovation rather than managing complex infrastructure. This flexibility, combined with the scalability and cost-efficiency of the cloud, makes it easier than ever to implement AI-driven solutions that can evolve alongside a business’s growth and needs. 3. Overcoming Challenges: Real-Time Data and AI Accuracy Real-Time Data Access For many SaaS applications, real-time data processing is crucial. AI-powered features need to respond to new inputs as they’re generated, providing immediate value to users. For instance, in customer support, AI must instantly interpret user queries and generate accurate, context-aware responses based on the latest data. Building a real-time data pipeline requires robust infrastructure, such as Apache Kafka or AWS Kinesis, to stream data as it’s created, ensuring that the SaaS product remains responsive and agile. Data Quality and Context The effectiveness of AI models depends on the quality and context of the data they process. Poor data quality can result in inaccurate predictions, a phenomenon often referred to as "hallucinations" in machine learning models. To mitigate this: Implement data validation systems to ensure data accuracy and relevance.Train AI models on context-aware data to improve prediction accuracy and generate actionable insights. 4. Scaling for Long-Term Success Building for Growth As SaaS products scale, so does the volume of data, which places additional demands on the data pipeline. To ensure that the pipeline can handle future growth, SaaS leaders should design their AI systems with scalability in mind. Cloud platforms like AWS, Google Cloud, and Azure offer scalable infrastructure to manage large datasets without the overhead of maintaining on-premise servers. Automation and Efficiency AI can also be leveraged to automate various aspects of the data pipeline, such as data cleansing, enrichment, and predictive analytics. Automation improves efficiency and reduces manual intervention, enabling teams to focus on higher-level tasks. Permissions & Security As the product scales, managing data permissions becomes more complex. Role-based access control (RBAC) and attribute-based access control (ABAC) systems ensure that only authorized users can access specific data sets. Additionally, implementing strong encryption protocols for both data at rest and in transit is essential to protect sensitive customer information. 5. Best Practices for SaaS Product Leaders Start Small, Scale Gradually While the idea of designing a fully integrated AI pipeline from the start can be appealing, it’s often more effective to begin with a focused, incremental approach. Start by solving specific use cases and iterating based on real-world feedback. This reduces risks and allows for continuous refinement before expanding to more complex tasks. Foster a Growth Mindset AI adoption in SaaS requires ongoing learning, adaptation, and experimentation. Teams should embrace a culture of curiosity and flexibility, continuously refining existing processes and exploring new AI models to stay competitive. Future-Proof Your Pipeline To ensure long-term success, invest in building a flexible, scalable pipeline that can adapt to changing needs and ongoing regulatory requirements. This includes staying updated on technological advancements, improving data security, and regularly revisiting your compliance strategies. 6. Conclusion Integrating AI into SaaS data pipelines is no longer optional — it’s a critical component of staying competitive in a data-driven world. From ensuring regulatory compliance to building scalable architectures, SaaS leaders must design AI systems that can handle real-time data flows, maintain high levels of accuracy, and scale as the product grows. By leveraging open-source tools, embracing automation, and building flexible pipelines that meet both operational and regulatory needs, SaaS companies can unlock the full potential of their data. This will drive smarter decision-making, improve customer experiences, and ultimately fuel sustainable growth. With the right strategy and mindset, SaaS leaders can turn AI-powered data pipelines into a significant competitive advantage, delivering greater value to customers while positioning themselves for future success.

By Venkata Gummadi

Exploring Binary Quantization in Vector Databases

Vector databases are specialized systems designed to handle the storage and retrieval of high-dimensional vector representations of unstructured, complex data — like images, text, or audio. By representing complex data as numerical vectors, these systems understand context and conceptual similarity, providing noticeably similar results to queries rather than exact matches, which enables advanced data analysis and retrieval. As the volume of data in vector databases increases, storing and retrieving information becomes increasingly challenging. Binary quantization simplifies high-dimensional vectors into compact binary codes, reducing data size and enhancing retrieval speed. This approach improves storage efficiency and enables faster searches, allowing databases to manage larger datasets more effectively. Understanding Binary Quantization After the initial embedding is obtained, Binary Quantization is then applied. Binary quantization reduces each feature of a given vector into a binary digit 0 or 1. It assigns 1 for positive values and 0 for negative values capturing the sign of the corresponding number. For example, if an image is represented by four distinct features where each feature holds a value in the range of FLOAT-32 storage units, performing binary quantization on this vector would convert each feature into a single binary digit. Thus, the original vector, which consists of four FLOAT-32 values, would be transformed into a vector with four binary digits, such as [1, 0,0, 1] occupying only 4 bits. This massively reduces the amount of space every vector takes by a factor of 32x by converting the number stored at every dimension from a float32 down to 1-bit. However, reversing this process is impossible — making this a lossy compression technique. Why Binary Quantization Works Well for High-Dimensional Data When locating a vector in space, the sign indicates the direction to move, while the magnitude specifies how far to move in that chosen direction. In binary quantization, we simplify the data by retaining the sign of each vector component — 1 for positive values and 0 for negative ones. While this might seem extreme, as it omits the magnitude of movement along each axis, it surprisingly works exceptionally well for high-dimensional vectors. Let's explore why this seemingly radical approach proves so effective! Advantages of Binary Quantization in Vector Databases Improved Performance Binary quantization enhances performance by representing vectors with binary codes (0s and 1s), allowing for the use of Hamming distance as a similarity metric. Hamming distance is computed using the XOR operation between binary vectors: XOR results in 1 where bits differ and 0 where they are the same. The number of 1s in the XOR result indicates the number of differing bits, providing a fast and efficient measure of similarity. This approach simplifies and speeds up vector comparisons compared to more complex distance metrics like Euclidean distance. Enhanced Efficiency Binary quantization compresses vectors from 32-bit floats to 1-bit binary digits, drastically reducing storage requirements, as illustrated in the figure above. This compression lowers storage costs and accelerates processing speeds, making it highly efficient for vector databases that need to store and manage huge amounts of data. Scalability We've already discussed how increasing dimensions reduces collisions in representation, which makes binary quantization even more effective for high-dimensional vectors. This enhanced capability allows for efficient management and storage of vast datasets since the compact binary format significantly reduces storage space and computational load. As the number of dimensions grows, the exponential increase in potential regions ensures minimal collisions, maintaining high performance and responsiveness. This makes it an excellent choice for scalable vector databases, capable of handling ever-growing data volumes with ease. Challenges and Considerations Accuracy and Precision While binary quantization significantly speeds up search operations, it impacts the accuracy and precision of the search results. Nuance and detail provided by higher-resolution data can be lost, leading to less precise results. Furthermore, Binary Quantization is lossy compression, meaning once the data has undergone quantization, the original information is irretrievably lost. Integrating binary quantization with advanced indexing techniques, such as HNSW, can help improve search accuracy while retaining the speed benefits of binary encoding. Implementation Complexity Specialized hardware and software like SIMD (Single Instruction, Multiple Data) instructions are essential for accelerating bitwise operations allowing multiple data points to be processed simultaneously, significantly speeding up computations even in a brute force approach for similarity calculation. Data Preprocessing Binary quantization assumes data to be in a normal distribution. When data is skewed or has outliers, binary quantization may lead to suboptimal results, affecting the accuracy and efficiency of the vector database. Metric Discrepancies The binary quantizer uses the Hamming distance accurately for angular-based metrics like cosine similarity but contradicts metrics like Euclidean distance. Thus, it should be properly selected according to the application domain to measure the distance between bits. Future Trends and Developments We can look forward to certain enhancements in binary quantization like the adjustment of thresholds based on data distribution to boost accuracy and incorporating feedback loops for continuous improvement. Additionally, combining binary quantization with advanced indexing techniques promises to further optimize search efficiency. Applications of Binary Quantization in Vector Databases Image and Video Retrieval: Images and videos represent high-dimensional data with substantial storage demands. For instance, a single high-resolution image can have millions of pixels, each requiring multiple bytes to represent color information. Binary quantization compresses these high-dimensional feature vectors into compact binary codes, significantly reducing storage needs and enhancing retrieval efficiency.Recommendation Systems: Binary quantization enhances recommendation systems by converting user and item feature vectors into compact binary codes, improving both speed and efficiency. This can be further optimized by combining with nearest neighbor techniques like LSH, ensuring accurate recommendations through refined searches.Natural Language Processing (NLP): Binary quantization aids in processing and analyzing textual data by reducing storage requirements in the vector database, enabling efficient performance. This NLP technique allows for faster retrieval and comparison of text data, making chatbots more responsive and effective in handling user queries. Conclusion Binary quantization offers a powerful solution for handling the complexities of high-dimensional vector data in vector databases. By converting high-dimensional vectors into compact binary codes, this technique drastically reduces storage requirements and accelerates retrieval times. Furthermore, its integration with advanced indexing methods can further enhance search accuracy and efficiency, making it a versatile tool in information retrieval. Vector databases used to store dimensional data can utilize fast storage hardware to accelerate your workload, whether AI training or RAG-based applications.

By Martin Summer

Spring Boot WebClient: Performance Optimization and Resilience

Spring WebClient is a reactive, non-blocking HTTP (HyperText Transfer Protocol) client designed for making requests to external services. It belongs to the Spring WebFlux framework and provides advanced, scalable handling of HTTP requests more efficiently than the RestTemplate. WebClient also supports parallel and reactive programming, making it suitable to perform a large volume of operations without blocking requests. It is ideal when you want to build high-performance applications, either by making external API calls or having thousands of concurrent requests. It improves debugging and blends well with Spring Boot and third-party products like Resilience4j and Spring Cloud. It is best suited for the development of superior, adaptable, and microservices running in the cloud. WebClient needs proper configuration to effectively deliver optimal performance and manage resource utilization and durability against short-term errors. The following guide covers advanced features for better tuning of WebClient. The following features are important for the improvement of the WebClient in microservices: Performance Improvement Configuring how many requests will be managed, also known as connection pooling, and setting timeouts for WebClient play a vital role in handling a large number of concurrent requests. By default, WebClient uses Reactor Netty, which calls for connection pooling and timeout options out of the box. A well-configured connection pool ensures that simultaneous requests are processed efficiently and safeguards against resource exhaustion. Enable SSL/TLS Validation WebClient can be configured to make secure connections between microservices by validating SSL/TLS client certificates. This configuration is essential when interacting with servers that require client certificates for mutual TLS authentication. Disable SSL Verification Disabling SSL verification is not a good practice as it can introduce security weaknesses. It should only be used for testing purposes. Add Resilience and Retry Mechanism to WebClient Circuit breakers are often used in distributed systems to stop cascading failures if a dependent service is down or not able to perform. Incorporating a circuit breaker filters bad requests and prevents them from affecting the rest of the application. Here, the Resilience4j library is being used as a circuit breaker, and WebClient is configured along with it. Transient errors may appear, which may result in failed HTTP requests. On this note, one can set up a retry mechanism so as to allow WebClient to automatically retry requests based on a policy. This will increase the likelihood of receiving a successful response in subsequent attempts. Enable Logging for HTTP Requests and Responses In microservices, one can get detailed logs, which may be useful for finding the root cause of actual issues with HTTP requests. The wiretap feature of WebClient can be used to log all request and response data. This is particularly useful when a root cause is required to be found. Serialization/Deserialization Configuration The built-in feature within WebClient has a powerful way of writing or reading a response in a specific data format. Custom serializer helps developers to implement their own serializers that can better handle the domain-specific data structures. Post this, WebClient helps in improving response time for microservices. Enable Error Handling WebClient makes it easier to handle HTTP errors due to the availability of HTTP status codes in response. It is displayed in the following example whether the status code is of type client error (4xx) or server error (5xx), allowing us to perform targeted error responses or retry mechanisms as needed. Default Header All HTTP requests in WebClient will have default headers. To avoid duplicated work for each request, it can be added globally. Compression GZIP compression provides higher performance, while microservices carry large request payloads. Conclusion Spring Boot's WebClient provides many features, such as connection pools, request timeouts, and retry policies, improving performance to a great extent. Circuit breaker functionality adds immunity to WebClient against failures. Error handling and compression features streamline data movement and reduce network usage. These configurations transform WebClient into a robust, high-performance HTTP client suitable for modern, reactive applications. By applying these strategies, applications will be better equipped to handle traffic surges, ensure fault tolerance, and provide a responsive experience for end-users. The code examples mentioned above are available in the GitHub repository. Further Reading To read more about optimizing Spring Boot’s WebClient for better performance, refer to the Spring Framework WebClient Documentation, and the broader insight is provided by the Spring Framework Reference Guide. Advanced netty configurations are documented in the Reactor Netty Documentation, while the Resilience4j Documentation elaborates on the circuit breakers and retry mechanisms. The common challenges and practical strategies are highlighted in Maximizing Performance with Netty and Reactive Programming in Java. Finally, the hands-on guide to integrating resilience patterns into spring boots is available in Baeldung's guide on Resilience4j. When used together, all these references provide a comprehensive guide to building scalable, efficient applications using WebClient.

By Sulakshana Singh

Addressing the Problem of Drift Detection and Drift Cause Analysis

Anyone working in DevOps today would likely agree that codifying resources makes it easier to observe, govern, and automate. However, most engineers would also acknowledge that this transformation brings with it a new set of challenges. Perhaps the biggest challenge of IaC operations is drifts — a scenario where runtime environments deviate from their IaC-defined states, creating a festering issue that could have serious long-term implications. These discrepancies undermine the consistency of cloud environments, leading to potential issues with infrastructure reliability and maintainability and even significant security and compliance risks. In an effort to minimize these risks, those responsible for managing these environments are classifying drift as a high-priority task (and a major time sink) for infrastructure operations teams. This has driven the growing adoption of drift detection tools that flag discrepancies between the desired configuration and the actual state of the infrastructure. While effective at detecting drift, these solutions are limited to issuing alerts and highlighting code diffs, without offering deeper insights into the root cause. Why Drift Detection Falls Short The current state of drift detection stems from the fact that drifts occur outside the established CI/CD pipeline and are often traced back to manual adjustments, API-triggered updates, or emergency fixes. As a result, these changes don’t usually leave an audit trail in the IaC layer, creating a blind spot that limits the tools to just flagging code discrepancies. This leaves platform engineering teams to speculate about the origins of drift and how it can best be addressed. This lack of clarity makes resolving drift a risky task. After all, automatically reverting changes without understanding their purpose — a common default approach — could be opening a can of worms and can trigger a cascade of issues. One risk is that this could undo legitimate adjustments or optimizations, potentially reintroducing problems that were already addressed or disrupting the operations of a valuable third-party tool. Take, for example, a manual fix applied outside the usual IaC process to address a sudden production issue. Before reverting such changes, it’s essential to codify them to preserve their intent and impact or risk prescribing a cure that could turn out to be worse than the illness. Detection Meets Context Seeing organizations grapple with these dilemmas has inspired the concept of ‘Drift Cause.’ This concept uses AI-assisted logic to sift through large event logs and provide additional context for each drift, tracing changes back to their origin — revealing not just ‘what’ but also ‘who,’ ‘when,’ and ‘why.’ This ability to process non-uniform logs in bulk and gather drift-related data flips the script on the reconciliation process. To illustrate, let me take you back to the scenario I mentioned earlier and paint a picture of receiving a drift alert from your detection solution — this time with added context. Now, with the information provided by Drift Cause, you can not only be aware of the drift but also zoom in to discover that the change was made by John at 2 a.m., right around the time the application was handling a traffic spike. Without this information, you might assume the drift is problematic and revert the change, potentially disrupting critical operations and causing downstream failures. With the added context, however, you get to connect the dots, reach out to John, confirm that the fix addressed an immediate issue, and decide that it shouldn’t be blindly reconciled. Moreover, using this context, you can also start thinking ahead and introduce adjustments to the configuration to add scalability and prevent the issue from recurring. This is a simple example, of course, but I hope it does well to show the benefit of having additional root cause context — an element long missing from drift detection despite being standard in other areas of debugging and troubleshooting. The goal, of course, is to help teams understand not just what changed but why it changed, empowering them to take the best course of action with confidence. Beyond IaC Management But having additional context for drift, as important as it may be, is only one piece of a much bigger puzzle. Managing large cloud fleets with codified resources introduces more than just drift challenges, especially at scale. Current-gen IaC management tools are effective at addressing resource management, but the demand for greater visibility and control in enterprise-scale environments is introducing new requirements and driving their inevitable evolution. One direction I see this evolution moving toward is Cloud Asset Management (CAM), which tracks and manages all resources in a cloud environment — whether provisioned via IaC, APIs, or manual operations — providing a unified view of assets and helping organizations understand configurations, dependencies, and risks, all of which are essential for compliance, cost optimization, and operational efficiency. While IaC management focuses on the operational aspects, Cloud Asset Management emphasizes visibility and understanding of cloud posture. Acting as an additional observability layer, it bridges the gap between codified workflows and ad-hoc changes, providing a comprehensive view of the infrastructure. 1+1 Will Equal Three The combination of IaC management and CAM empowers teams to manage complexity with clarity and control. As the end of the year approaches, it's 'prediction season' — so here’s mine. Having spent the better part of the last decade building and refining one of the more popular (if I may say so myself) IaC management platforms, I see this as the natural progression of our industry: combining IaC management, automation, and governance with enhanced visibility into non-codified assets. This synergy, I believe, will form the foundation for a better kind of cloud governance framework — one that is more precise, adaptable, and future-proof. By now, it’s almost a given that IaC is the bedrock of cloud infrastructure management. Yet, we must also acknowledge that not all assets will ever be codified. In such cases, an end-to-end infrastructure management solution can’t be limited to just the IaC layer. The next frontier, then, is helping teams expand visibility into non-codified assets, ensuring that as infrastructure evolves, it continues to perform seamlessly — one reconciled drift at a time and beyond.

By Omry Hay

Data Privacy and Security: A Developer's Guide to Handling Sensitive Data With DuckDB

Understanding DuckDB for Data Privacy and Security Data privacy and security have become critical for all organizations across the globe. Organizations often need to identify, mask, or remove sensitive information from their datasets while maintaining data utility. This article explores how to leverage DuckDB, an in-process analytical database, for efficient sensitive data remediation. Why DuckDB? (And Why Should You Care?) Think of DuckDB as SQLite's analytically gifted cousin. It's an embedded database that runs right in your process, but it's specifically designed for handling analytical workloads. What makes it perfect for data remediation? Well, imagine being able to process large datasets with lightning speed, without setting up a complicated database server. Sounds good, right? Here's what makes DuckDB particularly awesome for our use case: It's blazing fast thanks to its column-oriented storage.You can run it right in your existing Python environment.It handles multiple file formats like it's no big deal.It plays nicely with cloud storage (more on that later). In this guide, I'll be using Python along with DuckDB. DuckDB supports other languages, too, as mentioned in their documentation. Getting Started With DuckDB for Data Privacy Prerequisites Python 3.9 or higher installed Prior knowledge of setting up Python projects and virtual environments or Conda environments Install DuckDB inside a virtual environment by running the following command: Shell pip install duckdb --upgrade Now that you have installed DuckDB, let's create a DuckDB connection: Python import duckdb import pandas as pd # Create a DuckDB connection - it's this simple! conn = duckdb.connect(database=':memory:') Advanced PII Data Masking Techniques Here's how to implement robust PII (Personally Identifiable Information) masking: Let's say you've got a dataset full of customer information that needs to be cleaned up. Here's how you can handle common scenarios. Let's create sample data: SQL CREATE TABLE customer_data AS SELECT 'John Doe' as name, '123-45-6789' as ssn, 'john.doe@email.com' as email, '123-456-7890' as phone; This creates a table called customer_data with one row of sample-sensitive data.The data includes a name, SSN, email, and phone number. The second part involves masking patterns using regexp_replace: SQL -- Implement PII masking patterns CREATE TABLE masked_data AS SELECT regexp_replace(name, '[a-zA-Z]', 'X') as masked_name, regexp_replace(ssn, '[0-9]', '*') as masked_ssn, regexp_replace(email, '(^[^@]+)(@.*$)', '****$2') as masked_email, regexp_replace(phone, '[0-9]', '#') as masked_phone FROM customer_data; Let me walk you through what the above SQL code does. regexp_replace(name, '[a-zA-Z]', 'X') Replaces all letters (both uppercase and lowercase) with 'X'Example: "John Doe" becomes "XXXX XXX"regexp_replace(ssn, '[0-9]', '*') as masked_ssn Replaces all digits with '*'Example: "123-45-6789" becomes "--***"regexp_replace(email, '(^[^@]+)(@.*$)', '****$2') as masked_email: (^[^@]+) captures everything before the @ symbol(@.*$) captures the @ and everything after itReplaces the first part with '****' and keeps the domain partExample: "" becomes "****@email.com"regexp_replace(phone, '[0-9]', '#') as masked_phone: Replaces all digits with '#'Example: "123-456-7890" becomes "###-###-####" So your data is transformed as below: Original data: name: John Doe ssn: 123-45-6789 email: john.doe@email.com phone: 123-456-7890 Masked data: masked_name: XXXX XXX masked_ssn: ***-**-**** masked_email: ****@email.com masked_phone: ###-###-#### Python Implementation Python import duckdb import pandas as pd def mask_pii_data(): # Create a DuckDB connection in memory conn = duckdb.connect(database=':memory:') try: # Create and populate sample data conn.execute(""" CREATE TABLE customer_data AS SELECT 'John Doe' as name, '123-45-6789' as ssn, 'john.doe@email.com' as email, '123-456-7890' as phone """) # Implement PII masking conn.execute(""" CREATE TABLE masked_data AS SELECT regexp_replace(name, '[a-zA-Z]', 'X') as masked_name, regexp_replace(ssn, '[0-9]', '*') as masked_ssn, regexp_replace(email, '(^[^@]+)(@.*$)', '****$2') as masked_email, regexp_replace(phone, '[0-9]', '#') as masked_phone FROM customer_data """) # Fetch and display original data print("Original Data:") original_data = conn.execute("SELECT * FROM customer_data").fetchdf() print(original_data) print("\n") # Fetch and display masked data print("Masked Data:") masked_data = conn.execute("SELECT * FROM masked_data").fetchdf() print(masked_data) return original_data, masked_data except Exception as e: print(f"An error occurred: {str(e)}") return None, None finally: # Close the connection conn.close() Data Redaction Based on Rules Let me explain data redaction in simple terms before diving into its technical aspects. Data redaction is the process of hiding or removing sensitive information from documents or databases while preserving the overall structure and non-sensitive content. Think of it like using a black marker to hide confidential information on a printed document, but in digital form. Let's now implement Data Redaction with DuckDB and Python. I added this code snippet with comments so you can easily follow along. Python import duckdb import pandas as pd def demonstrate_data_redaction(): # Create a connection conn = duckdb.connect(':memory:') # Create sample data with various sensitive information conn.execute(""" CREATE TABLE sensitive_info AS SELECT * FROM ( VALUES ('John Doe', 'john.doe@email.com', 'CC: 4532-1234-5678-9012', 'Normal text'), ('Jane Smith', 'jane123@email.com', 'SSN: 123-45-6789', 'Some notes'), ('Bob Wilson', 'bob@email.com', 'Password: SecretPass123!', 'Regular info'), ('Alice Brown', 'alice.brown@email.com', 'API_KEY=abc123xyz', 'Basic text') ) AS t(name, email, sensitive_field, normal_text); """) # Define redaction rules redaction_rules = { 'email': r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}', # Email pattern 'sensitive_field': r'(CC:\s*\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}|SSN:\s*\d{3}-\d{2}-\d{4}|Password:\s*\S+|API_KEY=\S+)', # Various sensitive patterns 'name': r'[A-Z][a-z]+ [A-Z][a-z]+' # Full name pattern } # Show original data print("Original Data:") print(conn.execute("SELECT * FROM sensitive_info").fetchdf()) # Apply redaction redact_sensitive_data(conn, 'sensitive_info', redaction_rules) # Show redacted data print("\nRedacted Data:") print(conn.execute("SELECT * FROM redacted_data").fetchdf()) return conn def redact_sensitive_data(conn, table_name, rules): """ Redact sensitive data based on specified patterns. Parameters: - conn: DuckDB connection - table_name: Name of the table containing sensitive data - rules: Dictionary of column names and their corresponding regex patterns to match sensitive data """ redaction_cases = [] # This creates a CASE statement for each column # If the pattern matches, the value is redacted # If not, the original value is kept for column, pattern in rules.items(): redaction_cases.append(f""" CASE WHEN regexp_matches({column}, '{pattern}') THEN '(REDACTED)' ELSE {column} END as {column} """) query = f""" CREATE TABLE redacted_data AS SELECT {', '.join(redaction_cases)} FROM {table_name}; """ conn.execute(query) # Example with custom redaction patterns def demonstrate_custom_redaction(): conn = duckdb.connect(':memory:') # Create sample data conn.execute(""" CREATE TABLE customer_data AS SELECT * FROM ( VALUES ('John Doe', '123-45-6789', 'ACC#12345', '$5000'), ('Jane Smith', '987-65-4321', 'ACC#67890', '$3000'), ('Bob Wilson', '456-78-9012', 'ACC#11111', '$7500') ) AS t(name, ssn, account, balance); """) # Define custom redaction rules with different patterns custom_rules = { 'name': { 'pattern': r'[A-Z][a-z]+ [A-Z][a-z]+', 'replacement': lambda match: f"{match[0][0]}*** {match[0].split()[1][0]}***" }, 'ssn': { 'pattern': r'\d{3}-\d{2}-\d{4}', 'replacement': 'XXX-XX-XXXX' }, 'account': { 'pattern': r'ACC#\d{5}', 'replacement': 'ACC#*****' } } def apply_custom_redaction(conn, table_name, rules): redaction_cases = [] for column, rule in rules.items(): redaction_cases.append(f""" CASE WHEN regexp_matches({column}, '{rule['pattern']}') THEN '{rule['replacement']}' ELSE {column} END as {column} """) query = f""" CREATE TABLE custom_redacted AS SELECT {', '.join(redaction_cases)}, balance -- Keep this column unchanged FROM {table_name}; """ conn.execute(query) # Show original data print("\nOriginal Customer Data:") print(conn.execute("SELECT * FROM customer_data").fetchdf()) # Apply custom redaction apply_custom_redaction(conn, 'customer_data', custom_rules) # Show results print("\nCustom Redacted Data:") print(conn.execute("SELECT * FROM custom_redacted").fetchdf()) # Run demonstrations print("=== Basic Redaction Demo ===") demonstrate_data_redaction() print("\n=== Custom Redaction Demo ===") demonstrate_custom_redaction() Sample Results Before redaction: name email sensitive_field John Doe john.doe@email.com CC: 4532-1234-5678-9012 After redaction: name email sensitive_field (REDACTED) (REDACTED) (REDACTEd) Conclusion DuckDB is a simple, yet powerful in-memory database that can help with sensitive data remediation. Remember to always: Validate your masked data.Use parallel processing for large datasets.Take advantage of DuckDB's S3 integration for cloud data.Keep an eye on your memory usage when processing large files.

By Anil Kumar Moka

How to Build a YouTube Clone With Svelte.js, Firebase, and Permit.io

Creating a video-sharing application like YouTube is not just about front-end design and data storage; you need to have secure dynamic control over what users can see and do. With Svelte.js handling the interface and Firebase supporting backend functionalities, integrating Permit.io enables robust access control, using role-based access control (RBAC) to enforce detailed permissions. In this tutorial, you will build a secure YouTube clone that allows users to interact only within limited boundaries, according to their role and context. Building a YouTube Clone With RBAC Our goal is to create a YouTube clone where permissions control which users can upload, delete, or view videos based on the roles (RBAC) assigned to them. Create a Firebase App and Get Credentials Firebase is a hosted platform that helps people to create, run, and scale web and mobile apps. It offers services and tools such as authentication, real-time database, cloud storage, machine learning, remote configuration, and Static file hosting. We’ll use Firebase for our backend storage and authentication. Follow the steps below to set up your Firebase app: 1. Go to the Firebase Console and create a new project. 2. Enable Firestore Database, Firebase Authentication (Email/Password), and Firebase Storage to manage user data and videos. 3. After setting up your project, navigate to the project settings and add a new Web app. Enter App name as Youtube and click on Register app. 4. From your Firebase console, Click on the Settings icon → Project Settings → Service Account. Select Node.js and click on the “Generate new private key” button to download your credentials. With Firebase set up, we’re now ready to move on to our app build and integrate Permit.io for managing permissions. Understanding RBAC and How It Works Role-based access control (RBAC) is a method of managing user access to systems and resources based on a user's role or job responsibilities. With RBAC, permissions are easy to manage for different user types because roles determine what actions can be taken. Here is how it works in our app: Roles like Admin, Creator, and Viewer determine which actions users can take.Each API route checks the user’s role before allowing access. Let's look at this example on the flowchart below, which shows the user roles in our app and what access they have. Why Use RBAC? With RBAC, we can give permissions by setting roles such as Admin, Creator, and Viewer and giving each role what they can do. This structure also allows us to easily organize access into broad categories, which also means that if we have to change permissions for a group, it’s a quick role update. For example, Admins can have complete control over videos, Editors can edit the content, and Viewers can only view videos without any editing rights. To learn more about RBAC and why you should use it in your application, check out this blog. Setting Up Permit.io and Credentials To start with Permit.io, you’ll first need to set up an account and get the necessary credentials. These credentials are essential for connecting Permit.io to our backend, where we’ll handle all permission checks. To do that, follow the steps below: 1. Create a Permit.io account and set up a new project in the dashboard. 2. From your dashboard, click on Projects from the side panel. 3. By default, Permit.io provides you with two environments: Production and Development. Select your preferred environment or create a new one and Copy API Key. Creating RBAC Policies in Permit.io UI Now that we understand what RBAC is and how it applies to our YouTube clone app, let’s proceed; let’s create RBAC roles and policies in Permit.io. Follow these steps to set up RBAC: 1. In the Permit.io dashboard, click on Policy → Roles → Add Roles, and create admin, content_creator, and viewer roles. 2. From the policy page, click on Resources → Add Resource, create a video resource, and add the actions: create, read, delete, update, like and comment. 3. Navigate to the Policy Editor and grant each role access to the Video resource. From the above image, we have given the Admin full access to the Video resource and the Content Creator other access except to delete a video and the Viewer access to comment, create, read, and like a video. 1. After defining permissions for each role, save the policies. 2. Go to the Directory page and click the Add User button to create your first user. We have created a new user with a Viewer role. This means that this user will have all the access granted to the viewer role on the Video resource. Integrate Permit.io With the Backend Now that roles and permissions are defined, it’s time to connect Permit.io to our backend API. To get started quickly, clone the starter project and install dependencies. Plain Text git clone https://github.com/icode247/youtube_clone_starter cd youtube_clone_starter cd backend && npm install && cd .. && cd frontend && npm install This project includes the frontend built in Svelte, the backend API in Node.js, and Firebase integrations. Configuring Permit.io Next, create a config/permit.js file in your backend folder and add the Permit configuration: Plain Text import { Permit } from "permitio"; const permit = new Permit({ // We’ll use a cloud hosted policy decision point pdp: "https://cloudpdp.api.permit.io", token: process.env.PERMIT_API_KEY, }); export default permit; Create a .env file in the backend folder, and add the PERMIT_API_KEY you copied earlier. Plain Text PERMIT_API_KEY= Syncing Users With Permit.io To allow Permit.io to know our users and the roles they have, we need to sync our users with Permit.io at the point of successful registration. In controllers/auth/register.js file, update the register function with the code below to sync the user and assign a role after registration: Plain Text //... import permit from "../config/permit.js"; export async function register(req, res) { try { const { email, password, username, uid } = req.body; if (!email || !password || !username) { return res.status(400).json({ error: "Email, password, and username are required", }); } await db.collection("users").doc(uid).set({ email, username, createdAt: new Date().toISOString(), updatedAt: new Date().toISOString(), }); // Sync user with Permit.io await permit.api.syncUser({ key: email, email: email, first_name: username, last_name: "", attributes: {}, }); // Assign default viewer role await permit.api.assignRole( JSON.stringify({ user: email, role: "viewer", tenant: "default", }) ); res.status(201).json({ user: { id: uid, email: email, username: username, }, }); } catch (error) { console.error("Registration error:", error); if (error.code === "auth/email-already-exists") { return res.status(400).json({ error: "Email already in use", }); } res.status(500).json({ error: "Registration failed", }); } } //... In the above code, once a user registers, we sync it with Permit.io and assign it a Viewer role. Let’s register a new user named John Doe in our YouTube clone app and see our implementation in action. Go back to the Directory → Users tab, and you will see the new user we just created. Next, update the controllers/channelController.js file to assign a user a content-creator role when they create a channel so they can create and manage their videos. Plain Text //... export async function createChannel(req, res) { try { const { name, description } = req.body; const avatarFile = req.files?.avatarFile?.[0]; const bannerFile = req.files?.bannerFile?.[0]; let avatarUrl = null; if (avatarFile) { const avatarFileName = `channels/${req.user.uid}/avatar_${Date.now()}_${ avatarFile.originalname }`; const avatarRef = storage.bucket().file(avatarFileName); await avatarRef.save(avatarFile.buffer, { metadata: { contentType: avatarFile.mimetype, }, }); const [avatarSignedUrl] = await avatarRef.getSignedUrl({ action: "read", expires: "03-01-2500", }); avatarUrl = avatarSignedUrl; } let bannerUrl = null; if (bannerFile) { const bannerFileName = `channels/${req.user.uid}/banner_${Date.now()}_${ bannerFile.originalname }`; const bannerRef = storage.bucket().file(bannerFileName); await bannerRef.save(bannerFile.buffer, { metadata: { contentType: bannerFile.mimetype, }, }); const [bannerSignedUrl] = await bannerRef.getSignedUrl({ action: "read", expires: "03-01-2500", }); bannerUrl = bannerSignedUrl; } await db .collection("channels") .doc(req.user.uid) .set({ name, description, avatarUrl, bannerUrl, userName: name, createdAt: new Date().toISOString(), subscribers: 0, totalViews: 0, customization: { theme: "default", layout: "grid", }, }); const channelRef = db.collection("channels").doc(req.user.uid); const channelDoc = await channelRef.get(); //assign a user a new role await permit.api.assignRole({ user: req.user.email, role: "content_creator", tenant: "default", }); res.status(201).json({ id: channelRef.id, ...channelDoc.data(), }); } catch (error) { console.error("Error creating channel:", error); res.status(500).json({ error: "Failed to create channel" }); } } Now, go back to the app and click on the Create Channel button to create a new channel. So, after creating the channel, a new content_creator role will be added to the new users. Set Up Permission Middleware We’ve able to sync our users with Permit.io and assign them a role, let’s create a middleware to enforce the permissions and role checks we implemented. Create a newpermissions.js file in the middleware directory and add the code below to handle permission checks: Plain Text import permit from "../config/permit.js"; export const checkPermission = (action, resource) => async (req, res, next) => { const { email } = req.user; if (!email) { return res.status(401).json({ error: "Unauthorized" }); } try { const permitted = await permit.check(email, action, resource); if (!permitted) { return res.status(403).json({ error: "Forbidden" }); } next(); } catch (error) { console.error("Permission check failed:", error); res.status(500).json({ error: "Internal server error" }); } }; The checkPermission function takes two parameters: The action to be performed on the Video resourceThe resource on which the action will be performed on We then used the permit.check function from Permit.io to check whether the user has permission to access the resource. The first argument passed to the check function is the user's key, which we used the user’s email to sync the users with Permit.io, then the action and the resource. Applying Middleware in Routes Let’s update all our route files to use the permission middleware for protected routes. We’ll add to the routes/videos.js file, go ahead and add to other routes. Plain Text //... import { authenticateUser } from "../middleware/auth.js"; //... router.get("/", checkPermission("read", "video"), videoController.listVideos); router.get("/:id", checkPermission("read", "video"), videoController.getVideo); router.post( "/", checkPermission("create", "video"), uploadVideo, videoController.createVideo ); router.put( "/:id", checkPermission("update", "video"), videoController.updateVideo ); router.delete( "/:id", checkPermission("delete", "video"), videoController.deleteVideo ); router.post( "/:id/like", checkPermission("like", "video"), videoController.toggleLike ); router.get( "/:id/like", checkPermission("read", "video"), videoController.getLikes ); router.post( "/:id/comments", checkPermission("comment", "video"), videoController.addComment ); router.get( "/:id/comments", checkPermission("read", "video"), videoController.getComments ); //... Conclusion Implementing RBAC in our YouTube clone has provided a secure and manageable way to control user access based on roles. Permit.io’s simple UI and API make setting up and enforcing role-based permissions trivial, and we can easily assign and change access levels. This foundational layer of access control ensures that every user can interact with the app in their own way, based on their role, making the app ready for real-world use. For those exploring more advanced, context-sensitive access control, attribute-based access control (ABAC) is a logical next step, offering flexibility for future needs.

By Ekekenta Odionyenfe