DZone Spotlight

As the Trump administration revokes Executive Order 14110, the U.S. shifts toward a market-driven AI strategy, departing from the Biden administration’s regulatory framework. While proponents see this as a catalyst for innovation and economic growth, critics warn of increased risks, regulatory fragmentation, and strained transatlantic relations. With Europe reinforcing its AI Act and states like California exploring their own regulations, the future of AI governance in the U.S. remains uncertain. Will deregulation accelerate progress, or does it open the door to new challenges in ethics, security, and global cooperation? Just days after taking office, Donald Trump, the 47th President of the United States, issued a series of executive actions aimed at dismantling key initiatives from the Biden administration. Among them was the revocation of Executive Order (EO) 14110, a landmark policy that established a framework for AI governance and regulation. This decision marks a turning point in U.S. AI policy. For its supporters, it is a necessary reform; for its critics, it is a dangerous setback. While EO 14110 aimed to structure AI adoption by balancing innovation and oversight, its repeal raises critical questions about the future of AI in the United States and its global impact. Background on Executive Order 14110 Executive Order 14110 was issued on October 30, 2023, under the Biden administration. This major initiative aimed to regulate the development and deployment of artificial intelligence. Its goal was to balance innovation, security, and economic stability while ensuring that AI systems remained reliable, safe, and transparent. In the Biden administration’s vision, EO 14110 was designed to address key concerns such as algorithmic bias, misinformation, job displacement, and cybersecurity vulnerabilities. It was not intended to impose direct restrictions on the private sector but rather to establish security and ethical standards, particularly for AI used by federal agencies and in public sector contracts, while also influencing broader AI governance. From an international perspective, EO 14110 also aimed to strengthen the United States' role in global AI governance. It aligned with the European Union’s approach, particularly as the EU was developing its AI Act. The order was part of a broader transatlantic effort to establish ethical and security standards for AI. "Artificial Intelligence (AI) holds extraordinary potential for both promise and peril. Responsible AI use has the potential to help solve urgent challenges while making our world more prosperous, productive, innovative, and secure. At the same time, irresponsible use could exacerbate societal harms such as fraud, discrimination, bias, and disinformation; displace and disempower workers; stifle competition; and pose risks to national security." (EO 14110 - Section 1) EO 14110 as Part of a Broader AI Strategy: Continuity in Biden’s Policy It is important to understand that EO 14110 was not an isolated initiative. It was part of a broader strategy built on several existing frameworks and commitments. Blueprint for an AI Bill of Rights (2022). A foundational document outlining five key principles: safe and effective AI systems, protections against algorithmic discrimination, data privacy, transparency, and human alternatives. Voluntary AI Commitments (2023-2024). Major tech companies, including Google, OpenAI, and Microsoft, agreed to self-regulation measures focusing on AI transparency, security, and ethics. National Security AI Strategy (2024). The Biden administration made AI a priority in cybersecurity, military applications, and critical infrastructure protection. It is worth noting that even after the revocation of EO 14110, these initiatives remain in place, ensuring a degree of continuity in AI governance in the United States. Objectives and Scope of EO 14110 Executive Order 14110 pursued several strategic objectives aimed at regulating AI adoption while promoting innovation. It emphasized the security and reliability of AI systems by requiring robustness testing and risk assessments, particularly in sensitive areas such as cybersecurity and critical infrastructure. It also aimed to ensure fairness and combat bias by implementing protections against algorithmic discrimination and promoting ethical AI use in hiring, healthcare, and justice. EO 14110 included training, reskilling, and protection programs to help workers adapt to AI-driven changes. It also aimed to protect consumers by preventing fraudulent or harmful AI applications, ensuring safe and beneficial use. Finally, the executive order aimed to reinforce international cooperation, particularly with the European Union, to establish common AI governance standards. However, it’s important to note that it did not aim to regulate the entire private sector but rather to set strict ethical and security standards for AI systems used by federal agencies. Summary of EO 14110’s Key Principles To quickly get the essentials, here are the eight fundamental principles it was built on: Ensure AI is safe and secure.Promote responsible innovation and competition.Support workers affected by AI’s deployment.Advance equity and civil rights.Protect consumer interests in AI applications.Safeguard privacy and civil liberties.Enhance AI capabilities within the federal government.Promote U.S. leadership in global AI governance. Why Did the Trump Administration Revoke EO 14110? So, on January 20, 2025, the Trump administration announced the revocation of EO 14110, arguing that it restricted innovation by imposing excessive administrative constraints. The White House justified this decision as part of a broader push to deregulate the sector, boost the economy, and attract AI investment. The administration made clear its preference for a market-driven approach. According to Trump, private companies are better positioned to oversee AI development without federal intervention. Clearly, this shift marks a geopolitical turning point. The United States is moving away from a multilateral approach to assert its dominance in the AI sector. However, this revocation does not mean the end of AI regulation in the United States. Other federal initiatives, such as the NIST AI Risk Management Framework, remain in place. "Republicans support AI development rooted in free speech and human flourishing." (The 2024 Republican Party by Reuters) Consequences of the Revocation in the United States The repeal of EO 14110 has immediate effects and long-term implications. It reshapes the future of AI development in the United States. From the Trump administration’s perspective, this decision removes bureaucratic hurdles, accelerates innovation, and strengthens U.S. competitiveness in AI. Supporters argue that by reducing regulatory constraints, the repeal allows companies to move faster, lowers compliance costs, and attracts greater investment, particularly in automation and biotechnology. But on the other hand, without a federal framework, the risks associated with the development and use of AI technologies are increasing. Algorithmic bias, cybersecurity vulnerabilities, and the potential misuse of AI become harder to control without national oversight. Critics also warn of a weakening of worker and consumer protections, as the end of support programs could further deepen economic inequalities. In practical terms, regulation is becoming more fragmented. Without a federal framework, each state could, and likely will, develop its own AI laws, making compliance more complex for businesses operating nationwide. Some see this as an opportunity for regulatory experimentation, while others see it as a chance for opportunistic players to exploit loopholes or fear legal uncertainty and increased tensions with international partners. Impact on Europe The revocation of EO 14110 also affects global AI governance, particularly in Europe. Transatlantic relations are likely to become strained, as the growing divergence between U.S. and European approaches will make regulatory cooperation more challenging. European companies may tighten their compliance standards to maintain consumer trust, which could influence their strategic decisions. In fact, the European Union may face pressure to adjust its AI Act, although its regulatory framework remains largely independent from that of the United States. Conclusion The revocation of Executive Order 14110 is more than just a policy shift in the United States. It represents a strategic choice, favoring a deregulated model where innovation takes precedence over regulation. While this decision may help accelerate technological progress, it also leaves critical questions unanswered: Who will ensure the ethics, security, and transparency of AI in the United States? For Europe, this shift deepens the divide with the U.S. and strengthens its role as a "global regulator" through the AI Act. The European Union may find itself alone at the forefront of efforts to enforce strict AI regulations, risking a scenario where some companies favor the less restrictive U.S. market. More than a debate on regulation, this revocation raises a fundamental question: In the global AI race, should progress be pursued at all costs, or should every advancement be built on solid and ethical foundations? The choices made today will shape not only the future of the industry but also the role of democracies in the face of tech giants. One More Thing The revocation of EO 14110 highlights a broader debate: who really shapes AI policy, the government or private interests? While the U.S. moves toward deregulation, California’s AI safety bill (SB 1047) is taking the opposite approach, proposing strict oversight for advanced AI models. But as an investigation by Pirate Wires reveals, this push for regulation isn’t without controversy. Dan Hendrycks, a key advocate for AI safety, co-founded Gray Swan, a company developing compliance tools that could directly benefit from SB 1047’s mandates. This raises a crucial question: When policymakers and industry leaders are deeply intertwined, is AI regulation truly about safety, or about controlling the market? In the race to govern AI, transparency may be just as important as regulation itself.

By Frederic Jacquet CORE

Neon is now available on the Azure marketplace. The new integration between Neon and Azure allows you to manage your Neon subscription and billing through the Azure portal as if Neon were an Azure product. Azure serverless and Neon are a natural combination — Azure serverless frees you from managing your web server infrastructure. Neon does the same for databases, offering additional features like data branching and vector database extensions. That said, let's try out this new integration by building a URL shortener API with Neon, Azure serverless, and Node.js. Note: You should have access to a terminal, an editor like VS Code, and Node v22 or later installed. Setting Up the Infrastructure We are going to have to do things a little backward today. Instead of writing the code, we will first first set up our serverless function and database. Step 1. Open up the Azure web portal. If you don’t already have one, you will need to create a Microsoft account. Step 2. You will also have to create a subscription if you don’t have one already, which you can do in Azure. Step 3. Now, we can create a resource group to store our serverless function and database. Go to Azure's new resource group page and fill out the form like this: This is the Azure Resource Group creation page with the resource group set to "AzureNeonURLShortener" and the location set to West US 3.In general, use the location closest to you and your users, as the location will determine the default placement of serverless functions and what areas have the lowest latency. It isn’t vital in this example, but you can search through the dropdown if you want to use another. However, note that Neon doesn’t have locations in all of these regions yet, meaning you would have to place your database in a region further from your serverless function. Step 4. Click "Review & Create" at the bottom to access a configuration review page. Then click "Create" again. Step 5. Now, we can create a serverless function. Unfortunately, it includes another form. Go to the Azure Flex consumption serverless app creation page and complete the form. Use the resource group previously created, choose a unique serverless function name, place the function in your resource group region, and use Node v20. Step 6. The name you choose for your serverless app will be the subdomain Azure gives you to access your API, so choose wisely. After you finish filling everything out, click "Review and Create" and finally, "Create." Azure should redirect you to the new app's page. Now we can set up Neon. Open the new Neon Resource page on the Azure portal, and, you guessed it, fill out the form. How to Create a Neon Database on Azure Step 1. Create a new Neon resource page with "AzureURLNeonShortener" as the resource group, "URLShortenerDB" as the resource name, and "West US 3" as the location. If the area you chose isn’t available, choose the next closest region. Once you complete everything, click "Review & Create" and then "Create," as you did for previous resources. Step 2. You might have to wait a bit for the Neon database to instantiate. Once it does, open its configuration page and click "Go To Neon." Step 3. You will be redirected to a login page. Allow Neon to access your Azure information, and then you should find yourself on a project creation page. Fill out the form below: The project and database name aren't significant, but make sure to locate the database in Azure West US 3 (or whatever region you choose). This will prevent database queries from leaving the data center, decreasing latency. Step 4. Click "Create" at the bottom of the page, keeping the default autoscaling configuration. You should now be redirected to a Neon database page. This page has our connection string, which we will need to connect to our database from our code. Click "Copy snippet" to copy the connection string. Make sure you don’t lose this, as we will need it later, but for now, we need to structure our database. Step 5. Click “SQL Editor” on the side navigation, and paste the following SQL in: SQL CREATE TABLE IF NOT EXISTS urls(id char(12) PRIMARY KEY, url TEXT NOT NULL); Then click "Run." This will create the table we will use to store URLs. The table is pretty simple: The primary key ID is a 12 — character random string that we will use to refer to URLs, and the URL is a variable-length string that will store the URL itself. Step 6. If you look at the Table view on the side navigation, you should see a “urls” table. Finally, we need to get our connection string. Click on “Dashboard” on the side nav, find the connection string, and click “Copy snippet.” Now, we can start writing code. Building the API Step 1. First, we must install Azure’s serverless CLI, which will help us create a project and eventually test/publish it. Open a terminal and run: Plain Text npm install -g azure-functions-core-tools --unsafe-perm true Step 2. If you want to use other package managers like Yarn or pnpm, just replace npm with your preferred package manager. Now, we can start on our actual project. Open the folder you want the project to be in and run the following three commands: Plain Text func init --javascript func new --name submit --template "HTTP trigger" func new --name url --template "HTTP trigger" npm install nanoid @neondatabase/serverless Now, you should see a new Azure project in that folder. The first command creates the project, the two following commands create our serverless API routes, and the final command installs the Neon serverless driver for interfacing with our database and Nano ID for generating IDs. We could use a standard Postgres driver instead of the Neon driver, but Neon’s driver uses stateless HTTP queries to reduce latency for one-off queries. Because we are running a serverless function that might only process one request and send one query, one-off query latency is important. You will want to focus on the code in src/functions, as that is where our routes are. You should see two files there: submit.js and redirect.js. submit.js submit.js will store the code we use to submit URLs. First, open submit.js and replace its code with the following: TypeScript import { app } from "@azure/functions"; import { neon } from "@neondatabase/serverless"; import { nanoid } from "nanoid"; const sql = neon("[YOUR_POSTGRES_CONNECTION_STRING]"); app.http("submit", { methods: ["GET"], authLevel: "anonymous", route: "submit", handler: async (request, context) => { if (!request.query.get("url")) return { body: "No url provided", status: 400, }; if (!URL.canParse(request.query.get("url"))) return { body: "Error parsing url", status: 400, }; const id = nanoid(12); await sql`INSERT INTO urls(id,url) VALUES (${id},${request.query.get( "url" )})`; return new Response(`Shortened url created with id ${id}`); }, }); Let’s break this down step by step. First, we import the Azure functions API, Neon serverless driver, and Nano ID. We are using ESM (ES Modules) here instead of CommonJS. We will need to make a few changes later on to support this. Next, we create the connection to our database. Replace [YOUR_POSTGRES_CONNECTION_STRING] with the string you copied from the Neon dashboard. For security reasons, you would likely want to use a service like Azure Key Vault to manage your keys in a production environment, but for now, just placing them in the script will do. Now, we are at the actual route. The first few properties define when our route handler should be triggered: We want this route to be triggered by a GET request to submit. Our handler is pretty simple. We first check if a URL has been passed through the URL query parameter (e.g., /submit?url=https://google.com), then we check whether it is a valid URL via the new URL.canParse API. Next, We generate the ID with Nano ID. Because our IDs are 12 characters long, we have to pass 12 to the Nano ID generator. Finally, we insert a new row with the new ID and URL into our database. The Neon serverless driver automatically parameterizes queries, so we don’t need to worry about malicious users passing SQL statements into the URL. redirect.js redirect.js is where our actual URL redirects will happen. Replace its code with the following: TypeScript import { app } from "@azure/functions"; import { neon } from "@neondatabase/serverless"; const sql = neon("[YOUR_POSTGRES_CONNECTION_STRING]"); app.http("redirect", { methods: ["GET"], authLevel: "anonymous", route: "{id:length(12)}", handler: async (request, context) => { const url = await sql`SELECT * FROM urls WHERE urls.id=${request.params.id}`; if (!url[0]) return new Response("No redirect found", { status: 404 }); return Response.redirect(url[0].url, 308); }, }); The first section of the script is the same as submit.js. Once again, replace it \[YOUR\_POSTGRES\_CONNECTION\_STRING\] with the string you copied from the Neon dashboard. The route is where things get more interesting. We need to accept any path that could be a redirect ID, so we use a parameter with the constraint of 12 characters long. Note that this could overlap if you ever have another 12-character route. If it does, you can rename the redirect route to start with a Z or other alphanumerically greater character to make Azure serverless load the redirect route after. Finally, we have our actual handler code. All we need to do here is query for a URL matching the given ID and redirect to it if one exists. We use the 308 status code in our redirect to tell browsers and search engines to ignore the original shortened URL. Config Files There are two more changes we need to make. First, we don’t want a /api prefix on all our functions. To remove this, open host.json, which should be in your project directory, and add the following: TypeScript "extensions": { "http": { "routePrefix": "" } } This allows your routes to operate without any prefixes. The one other thing we need to do is switch the project to ES Modules. Open package.json and insert the following at the end of the file: Plain Text "type": "module" That’s it! Testing and Deploying Now, you can try testing locally by running func start. You can navigate to http://localhost:7071/submit?url=https://example.com, then use the ID it gives you and navigate to http://localhost:7071/[YOUR_ID]. You should be redirected to example.com. Of course, we can’t just run this locally. To deploy, we need to install the Azure CLI, which you can do with one of the following commands, depending on your operating system: macOS (Homebrew) Plain Text brew install azure-cli Windows (WPM) Plain Text winget install -e --id Microsoft.AzureCLI Linux Plain Text curl -L <https://aka.ms/InstallAzureCli> | bash Now, restart the terminal, log in by running az login, and run the following in the project directory: Plain Text func azure functionapp publish [FunctionAppName] Replace [FunctionAppName] with whatever you named your function earlier. Now, you should be able to access your API at [FunctionAppName].azurewebsites.net. Conclusion You should now have a fully functional URL Shortener. You can access the code here and work on adding a front end. If you want to keep reading about Neon and Azure’s features, we recommend checking out Branching. Either way, I hope you learned something valuable from this guide.

Overview One of the key principles of writing a good data pipeline is ensuring accurate data is loaded into the target table. We have no control over the quality of the upstream data we read from, but we can have a few data quality (DQ) checks in our pipeline to ensure any bad data would be caught early on without letting it propagate downstream. DQ checks are critical in making sure the data that gets processed every day is reliable, and that downstream tables can query them safely. This will save a lot of time and resources, as we will be able to halt the data flow, giving us some time to do RCA and fix the issue rather than pass incorrect data. The biggest challenge with large data warehouses with multiple interdependent pipelines is that we would have no idea about the data issue if bad data gets introduced in one of the pipelines, and sometimes, it could take days, even before it's detected. Even though DQ check failures could cause some temporary delay in landing the data, it's much better than customers or users reporting data quality issues and then having to backfill all the impacted tables. Some of the common data quality issues that could occur are: Duplicate rows – a table at user grain (which means there can only be one row per user), having duplicates0 or null values – you expect certain critical columns not to have any null or 0 values, e.g., SSN, age, country columnsAbnormal row count – the overall row count of the table suddenly increases or drops compared to the historical valuesAbnormal metric value – a specific metric, say '# of daily user logins' suddenly spikes or drops compared to historical values Note: The operators we will be referencing below are part of the Dataswarm and Presto tech stack, which are a proprietary data pipeline building tool and an SQL query engine, respectively, developed at Facebook. Importance of Signal Tables It's a good practice to publish signal tables, which should serve as the source for downstream pipelines. These are essentially linked views that can be created on top of any table. Since they are views, they don’t take up any storage, so there is no reason not to build them. These should be created only after the DQ checks pass, and downstream pipelines should wait for these signal tables to be available rather than waiting for the source table directly, as these would have been vetted for any data anomalies. Building the Right DAG In the data lineage flow below, if bad data gets loaded into table1, then without DQ checks, they would get passed on to table2 and table3 as there is no way for pipelines2 and 3 to know of any data issues, as all they do is simply check if the table1 data has landed. But if DQ checks had been implemented, then it would fail the job/pipeline, and the table1_sigal wouldn’t have been created; thus, the downstream WaitForOperators would still be waiting, stopping the propagation of bad data. Types of DQ Failures to Enforce Hard failure. If these DQ checks fail, the job will fail and notify the oncall or table owner, so the signal table will not be created. These could potentially cause downstream pipelines to be delayed and could be an issue if they have tighter Service Level Agreements (SLAs). But for critical pipelines, this might be worth it, as sending bad data could have catastrophic ripple effects.Soft failure. If these fail, the oncall and table owner would be notified, but the job won't fail, so the signal table would still get published, and the data would get loaded and propagated downstream. For cases where the data quality loss is tolerable, this can be used. Setting Up DQ Checks We will go over some examples of how we can set up the different DQ checks and some simplified trigger logic behind each of the DQ operators. Some things to know beforehand: '<DATEID>' is a macro that will resolve to the date the Dataswarm pipeline is scheduled to run (e.g., when the job runs on Oct 1, 2020, it will resolve to ds = '2020-10-01').The output of presto_api will be an array of dictionaries, e.g., [{'ds': '2020-10-01', 'userID': 123, ‘country’: ‘US’}, {'ds': '2020-10-01', 'userID': 124, ‘country’: ‘CA’}, {...}], where each dictionary value represents the corresponding row values of the table being queried, and the key is the column name. Below would be the table representation of the data, Duplicate Rows We can simply aggregate by the key column (e.g., userID) specified by the user and check if there are any duplicate rows present by peforming a simple GROUP BY with a HAVING clause, and limiting to just 1 row. The presto_results variable should be empty ([]); if not, then there are duplicates present in the table. Python # output will be an array of dict representing reach row in the table # eg [{'ds': '2020-10-01', 'userID': 123}, {...}] presto_results = presto_api( namespace = 'namespace_name', sql = ''' SELECT useriID FROM table WHERE ds = '<DATEID>' GROUP BY 1 HAVING SUM(1) > 1 LIMIT 1 ''' ) if len(presto_results) > 0: # NOTIFY oncall/owner # JOB KILLED else: # JOB SUCCESS 0 or Null Values We can check if any of the specified columns have any invalid values by leveraging count_if presto UDF. Here, the output, if there are no invalid values, should be [{'userid_null_count': 0}]. Python presto_results = presto_api( namespace = 'namespace_name', sql = ''' SELECT COUNT_IF( userid IS NULL OR userid = 0 ) AS userid_null_count FROM table WHERE ds = '<DATEID>' ''' ) if presto_results[0]['userid_null_count'] > 0: # NOTIFY oncall/owner # JOB KILLED else: # JOB SUCCESS Abnormal Row Count To get a sense of what the normal/expected row count is for a table on a daily basis, we can do a simple 7-day average of the previous 7 days, and if today's value deviates too much from that, we can trigger the alert. The thresholds can be either: Static – a fixed upper and lower threshold that is always static. Every day, the operator checks if today’s row count is either over or below the thresholds.Dynamic – use a +x% and -x% threshold value (you can start with, say, 15%, and adjust as needed), and if today's value is greater than the 7d avg + x% or lower than the 7d avg - x%, then trigger the alert. Python dq_insert_operator = PrestoInsertOperator( input_data = {"in": "source_table"}, output_data = {"out": "dq_check_output_table"}, select = """ SELECT SUM(1) AS row_count FROM source_table WHERE ds = '<DATEID>' """, ) dq_row_check_result = presto_api( namespace = 'namespace_name', sql = ''' SELECT ds, row_count FROM dq_check_output_table WHERE ds >= '<DATEID-7>' ORDER BY 1 ''' ) # we will loop through the dq_row_check_result object, which will have 8 values # where we will find the average between DATEID-7 and DATEID-1 and compare against DATEID x = .15 # threshold prev_7d_list = dq_row_check_result[0:7] prev_7d_sum = sum([prev_data['row_count'] for prev_data in prev_7d_list]) prev_7d_avg = prev_7d_sum/7 today_value = dq_row_check_result[-1]['row_count'] upper_threshold = prev_7d_avg * (1 + x) lower_threshold = prev_7d_avg * (1 - x) if today_value > upper_threshold or today_value < lower_threshold: # NOTIFY oncall/owner # JOB KILLED else: # JOB SUCCESS So, every day, we calculate the sum of the total row count and load it into a dq_check_output_table (a temporary intermediate table that is specially used for storing DQ aggregated results). Then, we query the last 7 days and today's data from that table and store the values in an object, which we then loop through to calculate the upper and lower thresholds and check if today's value is violating either of them. Abnormal Metric Value If there are specific metrics that you want to track to see if there are any anomalies, you can set them up similarly to the above 'abnormal row count' check. Python dq_insert_operator = PrestoInsertOperator( input_data={"in": "source_table"}, output_data={"out": "dq_check_output_table"}, select=""" SELECT APPROX_DISTINCT(userid) AS distinct_user_count, SUM(cost) AS total_cost, COUNT_IF(has_login = True) AS total_logins FROM source_table WHERE ds = '<DATEID>' """, ) dq_row_check_result = presto_api( namespace='namespace_name', sql=''' SELECT ds, distinct_user_count, total_cost, total_logins FROM table WHERE ds >= '<DATEID-7>' ORDER BY 1 ''' ) Here, we calculate the distinct_user_count, total_cost, and total_logins metric and load it into a dq_check_output_table table, which we will query to find the anomalies. Takeaways You can extend this to any kind of custom checks/alerts like month-over-month value changes, year-over-year changes, etc. You can also specify GROUP BY clauses, for example, track the metric value at the interface or country level over a period of time. You can set up a DQ check tracking dashboard, especially for important metrics, to see how they have been behaving over time. In the screenshot below, you can see that there have been DQ failures for two of the dates in the past, while for other days, it has been within the predefined range. This can also be used to get a sense of how stable the upstream data quality is. They can save a lot of time as developers would be able to catch issues early on and also figure out where in the lineage the issue is occurring.Sometimes, the alerts could be false positive (FP) (alerts generated not due to bad/incorrect data, but maybe due to seasonality/new product launch, there could be a genuine volume increase or decrease). We need to ensure such edge cases are handled correctly to avoid noisy alerts. There is nothing worse than oncall being bombarded with FP alerts, so we want to be mindful of the thresholds we set and tune them as needed periodically.

When it comes to managing infrastructure in the cloud, AWS provides several powerful tools that help automate the creation and management of resources. One of the most effective ways to handle deployments is through AWS CloudFormation. It allows you to define your infrastructure in a declarative way, making it easy to automate the provisioning of AWS services, including Elastic Beanstalk, serverless applications, EC2 instances, security groups, load balancers, and more. In this guide, we'll explore how to use AWS CloudFormation to deploy infrastructure programmatically. We'll also cover how to manually deploy resources via the AWS Management Console and how to integrate services like Elastic Beanstalk, serverless functions, EC2, IAM, and other AWS resources into your automated workflow. Using AWS CloudFormation for Infrastructure as Code AWS CloudFormation allows you to define your infrastructure using code. CloudFormation provides a unified framework to automate and version your infrastructure by setting up Elastic Beanstalk, EC2 instances, VPCs, IAM roles, Lambda functions, or serverless applications. CloudFormation templates are written in YAML or JSON format, and they define the resources you need to provision. With CloudFormation, you can automate everything from simple applications to complex, multi-service environments. Key Features of CloudFormation Declarative configuration. Describe the desired state of your infrastructure, and CloudFormation ensures that the current state matches it.Resource management. Automatically provisions and manages AWS resources such as EC2 instances, RDS databases, VPCs, Lambda functions, IAM roles, and more.Declarative stack updates. If you need to modify your infrastructure, simply update the CloudFormation template, and it will adjust your resources to the new desired state. Steps to Use CloudFormation for Various AWS Deployments Elastic Beanstalk Deployment With CloudFormation 1. Write a CloudFormation Template Create a YAML or JSON CloudFormation template to define your Elastic Beanstalk application and environment. This template can include resources like EC2 instances, security groups, scaling policies, and even the Elastic Beanstalk application itself. Example of CloudFormation Template (Elastic Beanstalk): YAML yaml Resources: MyElasticBeanstalkApplication: Type: 'AWS::ElasticBeanstalk::Application' Properties: ApplicationName: "my-application" Description: "Elastic Beanstalk Application for my React and Spring Boot app" MyElasticBeanstalkEnvironment: Type: 'AWS::ElasticBeanstalk::Environment' Properties: EnvironmentName: "my-app-env" ApplicationName: !Ref MyElasticBeanstalkApplication SolutionStackName: "64bit Amazon Linux 2 v3.4.9 running Docker" OptionSettings: - Namespace: "aws:autoscaling:asg" OptionName: "MaxSize" Value: "3" - Namespace: "aws:autoscaling:asg" OptionName: "MinSize" Value: "2" - Namespace: "aws:ec2:vpc" OptionName: "VPCId" Value: "vpc-xxxxxxx" - Namespace: "aws:ec2:vpc" OptionName: "Subnets" Value: "subnet-xxxxxxx,subnet-yyyyyyy" 2. Deploy the CloudFormation Stack Use the AWS CLI or AWS Management Console to deploy the CloudFormation stack. Once deployed, CloudFormation will automatically create all the resources defined in the template. Deploy via AWS CLI: YAML bash aws cloudformation create-stack --stack-name MyElasticBeanstalkStack --template-body file://my-template.yml Serverless Deployment With AWS Lambda, API Gateway, and DynamoDB CloudFormation is also great for deploying serverless applications. With services like AWS Lambda, API Gateway, DynamoDB, and S3, you can easily manage serverless workloads. 1. Create a Serverless CloudFormation Template This template will include a Lambda function, an API Gateway for accessing the function, and a DynamoDB table. Example of CloudFormation Template (Serverless): YAML yaml Resources: MyLambdaFunction: Type: 'AWS::Lambda::Function' Properties: FunctionName: "MyServerlessFunction" Handler: "index.handler" Role: arn:aws:iam::123456789012:role/lambda-execution-role Code: S3Bucket: "my-serverless-code-bucket" S3Key: "function-code.zip" Runtime: nodejs14.x MyAPIGateway: Type: 'AWS::ApiGateway::RestApi' Properties: Name: "MyAPI" Description: "API Gateway for My Serverless Application" MyDynamoDBTable: Type: 'AWS::DynamoDB::Table' Properties: TableName: "MyTable" AttributeDefinitions: - AttributeName: "id" AttributeType: "S" KeySchema: - AttributeName: "id" KeyType: "HASH" ProvisionedThroughput: ReadCapacityUnits: 5 WriteCapacityUnits: 5 2. Deploy the Serverless Stack Deploy your serverless application using the AWS CLI or AWS Management Console. YAML bash aws cloudformation create-stack --stack-name MyServerlessStack --template-body file://serverless-template.yml VPC and EC2 Deployment CloudFormation can automate the creation of a Virtual Private Cloud (VPC), subnets, security groups, and EC2 instances for more traditional workloads. 1. CloudFormation Template for VPC and EC2 This template defines a simple EC2 instance within a VPC, with a security group allowing HTTP traffic. Example of CloudFormation Template (VPC and EC2): YAML Resources: MyVPC: Type: 'AWS::EC2::VPC' Properties: CidrBlock: "10.0.0.0/16" EnableDnsSupport: "true" EnableDnsHostnames: "true" MySecurityGroup: Type: 'AWS::EC2::SecurityGroup' Properties: GroupDescription: "Allow HTTP and SSH traffic" SecurityGroupIngress: - IpProtocol: "tcp" FromPort: "80" ToPort: "80" CidrIp: "0.0.0.0/0" - IpProtocol: "tcp" FromPort: "22" ToPort: "22" CidrIp: "0.0.0.0/0" MyEC2Instance: Type: 'AWS::EC2::Instance' Properties: InstanceType: "t2.micro" ImageId: "ami-xxxxxxxx" SecurityGroupIds: - !Ref MySecurityGroup SubnetId: !Ref MyVPC 2. Deploy the Stack YAML aws cloudformation create-stack --stack-name MyEC2Stack --template-body file://vpc-ec2-template.yml Advanced Features of CloudFormation AWS CloudFormation offers more than just simple resource provisioning. Here are some of the advanced features that make CloudFormation a powerful tool for infrastructure automation: Stack Sets. Create and manage stacks across multiple AWS accounts and regions, allowing for consistent deployment of infrastructure across your organization.Change Sets. Before applying changes to your CloudFormation stack, preview the changes with a change set to ensure the desired outcome.Outputs. Output values from CloudFormation that you can use for other stacks or applications. For example, output the URL of an API Gateway or the IP address of an EC2 instance.Parameters. Pass in parameters to customize your stack without modifying the template itself, making it reusable in different environments.Mappings. Create key-value pairs for mapping configuration values, like AWS region-specific values, instance types, or other environment-specific parameters. Using CloudFormation With AWS Services Beyond Elastic Beanstalk CloudFormation isn't just limited to Elastic Beanstalk deployments — it's a flexible tool that can be used with a variety of AWS services, including: AWS Lambda. Automate the deployment of serverless functions along with triggers like API Gateway, S3, or DynamoDB events.Amazon S3. Use CloudFormation to create S3 buckets and manage their configuration.AWS IAM. Automate IAM role and policy creation to control access to your resources.Amazon RDS. Define RDS databases (MySQL, PostgreSQL, etc.) with all associated configurations like VPC settings, subnets, and security groups.Amazon SQS, SNS. Manage queues and topics for your application architecture using CloudFormation.Amazon ECS and EKS. Automate the creation and deployment of containerized applications with services like ECS and EKS. Manually Deploying Infrastructure from the AWS Management Console While CloudFormation automates the process, sometimes manual intervention is necessary. The AWS Management Console allows you to deploy resources manually. 1. Elastic Beanstalk Application Go to the Elastic Beanstalk Console.Click Create Application, follow the steps to define the application name and platform (e.g., Docker, Node.js), and then manually configure the environment, scaling, and security options. 2. Serverless Applications (Lambda + API Gateway) Go to Lambda Console to create and deploy functions.Use API Gateway Console to create APIs for your Lambda functions. 3. EC2 Instances Manually launch EC2 instances from the EC2 Console and configure them with your chosen instance type, security groups, and key pairs. Conclusion AWS CloudFormation provides a consistent and repeatable way to manage infrastructure for Elastic Beanstalk applications, serverless architectures, and EC2-based applications. With its advanced features like Stack Sets, Change Sets, and Parameters, CloudFormation can scale to meet the needs of complex environments. For anyone managing large or dynamic AWS environments, CloudFormation is an essential tool for ensuring consistency, security, and automation across all your AWS deployments.

Keycloak is a powerful authentication and authorization solution that provides plenty of useful features, such as roles and subgroups, an advanced password policy, and single sign-on. It’s also very easy to integrate with other solutions. We’ve already shown you how to connect Keycloak to your Angular app, but there’s more you can do. For example, by integrating this technology with Cypress, you can enable the simulation of real-user login scenarios, including multi-factor authentication and social logins, ensuring that security protocols are correctly implemented and functioning as expected. Most importantly, you can also use Docker containers to provide a portable and consistent environment across different platforms (possibly with container image scanning, for increased security). This integration ensures easy deployment, scalability, and efficient dependency management, streamlining the process of securing applications and services. Additionally, Docker Compose can be used to orchestrate multiple containers, simplifying complex configurations and enhancing the overall management of Keycloak instances. This guide will show you precisely how to set all of this up. Let’s get started! Prerequisites The article is based on the contents of a GitHub repository consisting of several elements: Frontend application written in AngularKeycloak configurationE2E tests written in CypressDocker configuration for the whole stack The point of this tech stack is to allow users to work with Angular/Keycloak/Cypress locally and also in Docker containers. Keycloak Configuration We’ll start by setting up Keycloak, which is a crucial part of both configurations. The idea is to run it inside a Docker container and expose it at http://localhost:8080. Keycloak has predefined configurations, including users, realm, and client ID, so setting it up for this project requires minimum effort. Normal User Your normal user in the Keycloak panel should be configured using the following details: User: testPassword: sIjKqg73MTf9uTU Keycloak Administrator Here’s the default configuration for the admin user (of course, you probably shouldn’t use default settings for the admin account in real-world scenarios). User: adminPassword: admin Local Configuration This configuration allows you to work locally with an Angular application in dev mode along with E2E tests. It requires Keycloak to be run and available on http://localhost:8080. This is set in the Docker configuration, which is partially used here. To run the configuration locally, use the following commands in the command line. First, in the main project directory: JavaScript npm install In /e2e directory: JavaScript npm install In the main directory for frontend application development: JavaScript npm run start In /e2e directory: JavaScript npm run cy:run In the main project directory: JavaScript docker-compose up -d keycloak Docker Configuration Installing and configuring Docker is a relatively simple matter — the solution provides detailed documentation you can use if you run into any problems. In the context of our project, the Docker configuration does several key things: Running Keycloak and importing the predefined realm along with usersBuilding and exposing the Angular application on http://localhost:4200 via nginx on a separate Docker containerRunning e2e container to allow you to run tests via Cypress To run a dockerized configuration, type in the command line in the main project directory: JavaScript docker-compose up -d To run Cypress tests inside the container, use the following command: JavaScript docker container exec -ti e2e bash Then, inside the container, run: JavaScript npm run cy:run Test artifacts are connected to the host machine via volume, so test reports, screenshots, and videos will be available immediately on path /e2e/cypress/ in the following folders: reports, screenshots, and videos. Conclusion And that’s about it. As you can see, integrating Keycloak (or rather an Angular app that uses Keycloak), Docker, and Cypress is a relatively straightforward process. There are only a couple of steps you must take to get a consistent, containerized environment for easy deployment, scaling, and efficient dependency management — with the added benefit of real-user login scenario simulation thanks to Cypress for top-notch security.

DuckDb is a powerful in-memory database that has a parallel processing feature, which makes it a good choice to read/transform cloud storage data, in this case, AWS S3. I've had a lot of success using it and I will walk you through the steps in implementing it. I will also include some learnings and best practices for you. Using the DuckDb, httpfs extension and pyarrow, we can efficiently process Parquet files stored in S3 buckets. Let's dive in: Before starting the installation of DuckDb, make sure you have these prerequisites: Python 3.9 or higher installed Prior knowledge of setting up Python projects and virtual environments or conda environments Installing Dependencies First, let's establish the necessary environment: Shell # Install required packages for cloud integration pip install "duckdb>=0.8.0" pyarrow pandas boto3 requests The dependencies explained: duckdb>=0.8.0: The core database engine that provides SQL functionality and in-memory processingpyarrow: Handles Parquet file operations efficiently with columnar storage supportpandas: Enables powerful data manipulation and analysis capabilitiesboto3: AWS SDK for Python, providing interfaces to AWS servicesrequests: Manages HTTP communications for cloud interactions Configuring Secure Cloud Access Python import duckdb import os # Initialize DuckDB with cloud support conn = duckdb.connect(':memory:') conn.execute("INSTALL httpfs;") conn.execute("LOAD httpfs;") # Secure AWS configuration conn.execute(""" SET s3_region='your-region'; SET s3_access_key_id='your-access-key'; SET s3_secret_access_key='your-secret-key'; """) This initialization code does several important things: Creates a new DuckDB connection in memory using :memory:Installs and loads the HTTP filesystem extension (httpfs) which enables cloud storage accessConfigures AWS credentials with your specific region and access keysSets up a secure connection to AWS services Processing AWS S3 Parquet Files Let's examine a comprehensive example of processing Parquet files with sensitive data masking: Python import duckdb import pandas as pd # Create sample data to demonstrate parquet processing sample_data = pd.DataFrame({ 'name': ['John Smith', 'Jane Doe', 'Bob Wilson', 'Alice Brown'], 'email': ['john.smith@email.com', 'jane.doe@company.com', 'bob@email.net', 'alice.b@org.com'], 'phone': ['123-456-7890', '234-567-8901', '345-678-9012', '456-789-0123'], 'ssn': ['123-45-6789', '234-56-7890', '345-67-8901', '456-78-9012'], 'address': ['123 Main St', '456 Oak Ave', '789 Pine Rd', '321 Elm Dr'], 'salary': [75000, 85000, 65000, 95000] # Non-sensitive data }) This sample data creation helps us demonstrate data masking techniques. We include various types of sensitive information commonly found in real-world datasets: Personal identifiers (name, SSN)Contact information (email, phone, address)Financial data (salary) Now, let's look at the processing function: Python def demonstrate_parquet_processing(): # Create a DuckDB connection conn = duckdb.connect(':memory:') # Save sample data as parquet sample_data.to_parquet('sample_data.parquet') # Define sensitive columns to mask sensitive_cols = ['email', 'phone', 'ssn'] # Process the parquet file with masking query = f""" CREATE TABLE masked_data AS SELECT -- Mask name: keep first letter of first and last name regexp_replace(name, '([A-Z])[a-z]+ ([A-Z])[a-z]+', '\1*** \2***') as name, -- Mask email: hide everything before @ regexp_replace(email, '([a-zA-Z0-9._%+-]+)(@.*)', '****\2') as email, -- Mask phone: show only last 4 digits regexp_replace(phone, '[0-9]{3}-[0-9]{3}-', '***-***-') as phone, -- Mask SSN: show only last 4 digits regexp_replace(ssn, '[0-9]{3}-[0-9]{2}-', '***-**-') as ssn, -- Mask address: show only street type regexp_replace(address, '[0-9]+ [A-Za-z]+ ', '*** ') as address, -- Keep non-sensitive data as is salary FROM read_parquet('sample_data.parquet'); """ Let's break down this processing function: We create a new DuckDB connectionConvert our sample DataFrame to a Parquet fileDefine which columns contain sensitive informationCreate a SQL query that applies different masking patterns: Names: Preserves initials (e.g., "John Smith" → "J*** S***")Emails: Hides local part while keeping domain (e.g., "" → "****@email.com")Phone numbers: Shows only the last four digitsSSNs: Displays only the last four digitsAddresses: Keeps only street typeSalary: Remains unmasked as non-sensitive data The output should look like: Plain Text Original Data: ============= name email phone ssn address salary 0 John Smith john.smith@email.com 123-456-7890 123-45-6789 123 Main St 75000 1 Jane Doe jane.doe@company.com 234-567-8901 234-56-7890 456 Oak Ave 85000 2 Bob Wilson bob@email.net 345-678-9012 345-67-8901 789 Pine Rd 65000 3 Alice Brown alice.b@org.com 456-789-0123 456-78-9012 321 Elm Dr 95000 Masked Data: =========== name email phone ssn address salary 0 J*** S*** ****@email.com ***-***-7890 ***-**-6789 *** St 75000 1 J*** D*** ****@company.com ***-***-8901 ***-**-7890 *** Ave 85000 2 B*** W*** ****@email.net ***-***-9012 ***-**-8901 *** Rd 65000 3 A*** B*** ****@org.com ***-***-0123 ***-**-9012 *** Dr 95000 Now, let's explore different masking patterns with explanations in the comments of the Python code snippets: Email Masking Variations Python # Show first letter only "john.smith@email.com" → "j***@email.com" # Show domain only "john.smith@email.com" → "****@email.com" # Show first and last letter "john.smith@email.com" → "j*********h@email.com" Phone Number Masking Python # Last 4 digits only "123-456-7890" → "***-***-7890" # First 3 digits only "123-456-7890" → "123-***-****" # Middle digits only "123-456-7890" → "***-456-****" Name Masking Python # Initials only "John Smith" → "J.S." # First letter of each word "John Smith" → "J*** S***" # Fixed length masking "John Smith" → "XXXX XXXXX" Efficient Partitioned Data Processing When dealing with large datasets, partitioning becomes crucial. Here's how to handle partitioned data efficiently: Python def process_partitioned_data(base_path, partition_column, sensitive_columns): """ Process partitioned data efficiently Parameters: - base_path: Base path to partitioned data - partition_column: Column used for partitioning (e.g., 'date') - sensitive_columns: List of columns to mask """ conn = duckdb.connect(':memory:') try: # 1. List all partitions query = f""" WITH partitions AS ( SELECT DISTINCT {partition_column} FROM read_parquet('{base_path}/*/*.parquet') ) SELECT * FROM partitions; """ This function demonstrates several important concepts: Dynamic partition discoveryMemory-efficient processingError handling with proper cleanupMasked data output generation The partition structure typically looks like: Partition Structure Plain Text sample_data/ ├── date=2024-01-01/ │ └── data.parquet ├── date=2024-01-02/ │ └── data.parquet └── date=2024-01-03/ └── data.parquet Sample Data Plain Text Original Data: date customer_id email phone amount 2024-01-01 1 user1@email.com 123-456-0001 500.00 2024-01-01 2 user2@email.com 123-456-0002 750.25 ... Masked Data: date customer_id email phone amount 2024-01-01 1 **** **** 500.00 2024-01-01 2 **** **** 750.25 Below are some benefits of partitioned processing: Reduced memory footprintParallel processing capabilityImproved performanceScalable data handling Performance Optimization Techniques 1. Configuring Parallel Processing Python # Optimize for performance conn.execute(""" SET partial_streaming=true; SET threads=4; SET memory_limit='4GB'; """) These settings: Enable partial streaming for better memory managementSet parallel processing threadsDefine memory limits to prevent overflow 2. Robust Error Handling Python def robust_s3_read(s3_path, max_retries=3): """ Implement reliable S3 data reading with retries. Parameters: - s3_path: Path to S3 data - max_retries: Maximum retry attempts """ for attempt in range(max_retries): try: return conn.execute(f"SELECT * FROM read_parquet('{s3_path}')") except Exception as e: if attempt == max_retries - 1: raise time.sleep(2 ** attempt) # Exponential backoff This code block demonstrates how to implement retries and also throw exceptions where needed so as to take proactive measures. 3. Storage Optimization Python # Efficient data storage with compression conn.execute(""" COPY (SELECT * FROM masked_data) TO 's3://output-bucket/masked_data.parquet' (FORMAT 'parquet', COMPRESSION 'ZSTD'); """) This code block demonstrates applying storage compression type for optimizing the storage. Best Practices and Recommendations Security Best Practices Security is crucial when handling data, especially in cloud environments. Following these practices helps protect sensitive information and maintain compliance: IAM roles. Use AWS Identity and Access Management roles instead of direct access keys when possibleKey rotation. Implement regular rotation of access keysLeast privilege. Grant minimum necessary permissionsAccess monitoring. Regularly review and audit access patterns Why it's important: Security breaches can lead to data leaks, compliance violations, and financial losses. Proper security measures protect both your organization and your users' data. Performance Optimization Optimizing performance ensures efficient resource utilization and faster data processing: Partition sizing. Choose appropriate partition sizes based on data volume and processing patternsParallel processing. Utilize multiple threads for faster processingMemory management. Monitor and optimize memory usageQuery optimization. Structure queries for maximum efficiency Why it's important: Efficient performance reduces processing time, saves computational resources, and improves overall system reliability. Error Handling Robust error handling ensures reliable data processing: Retry mechanisms. Implement exponential backoff for failed operationsComprehensive logging. Maintain detailed logs for debuggingStatus monitoring. Track processing progressEdge cases. Handle unexpected data scenarios Why it's important: Proper error handling prevents data loss, ensures processing completeness, and makes troubleshooting easier. Conclusion Cloud data processing with DuckDB and AWS S3 offers a powerful combination of performance and security. Let me know how your DuckDb implementation goes!error handling

A while ago, I explored Neo4j, a powerful graph database, earlier for the implementation of the "six degrees of separation" concept. I like the way it simplified the complexity of the relationship visually with the advantage of flexible schema representation. After some time, while exploring another use case in Identity Access Management (IAM) policies, I revisited Neo4j. I was amazed by the analytical capabilities it offered and the potential for building Generative AI applications using Neo4j’s GraphRAG. In this article, we will see how we can use Neo4j for IAM data modeling, its use cases, guidelines, and best practices, and how Neo4j can be integrated with existing IAM systems and tools. Identity Access Management Identity Access Management (IAM) doesn’t need an introduction or a justification for its need, with so much happening in the field of cybersecurity. Ensuring authorized individuals, groups, and systems have appropriate access to the right resources at the right times. The key pillars of IAM, Access Management, Identity Governance and Administration, and Active Directory Management, are not a luxury anymore. Rather, they have become the key necessities of any organization looking forward to managing effective access management. Neo4J Neo4j is a powerful graph database that stores data and represents the relationship between data objects in a flexible graph structure. What I personally liked about Neo4j is the visual representation of complex scenarios. This helps in understanding simple yet powerful concepts of nodes, relationships, and attributes to depict, analyze, and implement complex use cases. Neo4j can be effectively utilized to model, analyze, and enforce IAM policies. It helps an organization improve IAM compliance with adherence and audits of the implemented policies. Why Neo4j for IAM? There is so much in common between the concepts of IAM and what Neo4j has to offer. The use case on fraud detection helped me understand and leverage the concepts for IAM. Representation of IAM Entities Using Graph Model Initially, I explored this for the concept of "six degrees of separation," where the relationships can be visually represented. Similarly, IAM involves complex relationships between users, roles, groups, resources, and access permissions. Neo4j's native graph model simplifies this complexity by directly representing entities as nodes and their associations as edges. We will see the representation in detail under the "Data Modelling" section. Real-Time Query Capabilities To be able to verify and validate access rights, user hierarchies, group memberships, and other critical access privileges, Neo4j provides real-time query capabilities. Agility The changes in roles, policies, and resources can be easily accommodated by Neo4j's schema-less model. It can also be used with a schema when needed. This flexibility is critical as the nature of IAM is so dynamic to be able to handle cyber threats and other challenges effectively. Analytical Capabilities As part of audit and compliance, it is important to be able to identify orphaned accounts, notify excessive permissions, and detect policy violations. Neo4j’s Cypher query language provides analytical capabilities to perform such complex tasks. Data Modeling in Neo4j for IAM We all know how crucial it is to have efficient and effective Data Modelling for any architecture. Neo4j leverages the concepts of graph theory, such as Vertices (also known as nodes), Edges (also known as relationships), and data points (also known as attributes). Below are some examples of how nodes, relationships, and attributes can be represented in IAM. Nodes User – represents an individual or service accountGroup – represents collections of usersRole – represents a set of permissionsResource – represents a system, file, or service that requires access controlPermission – represents access rights (e.g., READ, WRITE, EXECUTE) Relationships (User)-[:MEMBER_OF]->(Group)(User)-[:ASSIGNED]->(Role)(Group)-[:ASSIGNED]->(Role)(Role)-[:GRANTS]->(Permission)(Permission)-[:APPLIES_TO]->(Resource) Attributes Each node and relationship can have attributes like timestamps, descriptions, and status flags for audit and compliance tracking. A sample schema is given that depicts two scenarios: When a user is assigned to a role: Plain Text (User)-[:ASSIGNED]->(Role) When a user who is part of a certain group assigned to a role grants certain permission that applies to a certain resource: Plain Text (User)-[:MEMBER_OF]->(Group)-[:ASSIGNED]->(Role)-[:GRANTS]->(Permission)-[:APPLIES_TO]->(Resource) I can’t think of any use case of IAM that may not be able to be implemented by Neo4j. In the next section, we will see some of them. Use Cases Access Control This refers to being able to understand if a user has access to a certain resource. Query: "Does user X have access to resource Y?" Cypher query example: Plain Text MATCH (u:User {id: 'userX'})-[:ASSIGNED|MEMBER_OF*]->(r:Role)-[:GRANTS]->(p:Permission)-[:APPLIES_TO]->(res:Resource {id: 'resourceY'}) RETURN p Compliance Audits This refers to identifying users or roles with access to sensitive resources. Also, we can detect permission assignments that violate the policy of the organization. It can help analyze role hierarchies and remove redundant roles. Plain Text MATCH (r:Role)-[:GRANTS]->(p:Permission) WITH r, COUNT(p) as permissionCount RETURN r, permissionCount ORDER BY permissionCount DESC It can detect orphaned accounts with the query to find users not assigned to any group or role. Plain Text MATCH (u:User) WHERE NOT (u)-[:MEMBER_OF|ASSIGNED]->(:Group|:Role) RETURN u Access Hierarchy This addresses one of the complex queries on "What are all the resources user X can access through group Y?" Cypher query example: Plain Text MATCH (u:User {id: 'userX'})-[:MEMBER_OF]->(g:Group {id: 'groupY'})-[:ASSIGNED]->(r:Role)-[:GRANTS]->(p:Permission)-[:APPLIES_TO]->(res:Resource) RETURN res It is important to understand the guidelines and best practices to implement Neo4j effectively for IAM. Listing below some of the guidelines: Guidelines and Best Practices Security. Neo4j provides built-in role-based access control (RBAC), which can be leveraged to ensure secure access to the graph database.Indexing. To be able to optimize query performance, the recommendation is to use indexes on frequently queried properties like User.id, Group.id, and Resource.idIngestion. Since the data ingestion can impact the performance and load, the suggestion is to use batch imports or streaming tools like Neo4j’s ETL or Kafka Connectors for efficient data loading.Monitoring. Despite the effective architecture & implementation, over a while, due to various reasons, the performance may degrade. Below are some of the recommendations. For visualization and tracking performance, Neo4j’s monitoring tools (e.g., Neo4j Bloom, Prometheus) can be leveraged.As part of regular security audits and maintenance, the nodes and relationships are to be validated. An organization can have multiple IAM systems and tools to strengthen its security posture. The below section details how Neo4j can be integrated with existing systems and tools. Integration Neo4j can be integrated with various IAM systems and tools, such as: OAuth and OpenID connect. To model and analyze token-based access, Neo4j can be used.Active directory. To achieve centralized access control, the user and group data can be imported.Security Information and Event Management (SIEM) tools. In one of my articles, I have detailed how Elastic can be used as an effective SIEM tool. Neo4j queries and analytics can be fed into such systems for real-time monitoring.Custom APIs. To enable IAM functionality in applications, Neo4j’s drivers can be used to build APIs using Neo4j’s drivers (e.g., Java, Python, or JavaScript). Please refer to Neo4j’s documentation on creating applications for further details. Summary As the complexity and challenging requirements of Identity Access Management increase, we need a system such as Neo4j that provides a robust, flexible, and scalable. The visual representation and graphical analytical capabilities are particularly well-suited for modeling relationships, real-time visibility, enhancing compliance, and optimizing IAM policies and processes of an organization.

By Ramesh Manickavel CORE

The release of the DeepSeek open-source AI model has created a lot of excitement in the technology community. It allows developers to build applications entirely locally without needing to connect to online AI models such as Claude, ChatGPT, and more. Open-source models have opened doors to new opportunities when building enterprise applications that integrate with generative AI. In this article, you will learn how to run such a model locally on your personal machine and build a full-stack React and NodeJS-powered application that is not just another chatbot. You will be able to use this application to analyze resumes faster and make smarter hiring decisions. Before you build the application, it is important to understand the benefits of open-source LLMs. Benefits of Open-Source Large Language Models Open-source models provide several key benefits over using proprietary models : Cost-Effective and License-Free Open-source LLMs are cost-effective and don't need a special license. For example, as of the date of writing this, OpenAI’s o1 costs $60 per million output tokens, and open-source DeepSeek R1 costs $2.19. Customizable and Fine-Tunable Open-source models can be fine-tuned easily to meet unique business cases - allowing for more domain-specific use cases to be built. This leads to optimized performance in the enterprise applications. Enhanced Data Security and Privacy Open source makes applications more secure as precious personal data doesn't need to be uploaded to third-party servers and stays on a local machine or a company's network only stay in local machine or a companies network only. This provides a high level of data security. Furthermore, open-source models can be fine-tuned to remove any biases of data. Community-Driven and No Vendor Lock-In Open-source models enjoy large community support and benefit from the rapid pace of feature development. On the other hand, using property models makes the application vendor-locked and reliant on vendor companies to provide feature updates. With this information in hand, you are ready to build a real-world application using the DeepSeek R1 open-source model, Node.JS, and React. Project and Architecture Overview You will be building a resume analyzer application — which will help you learn the benefits and shortcomings of the uploaded resume. DeepSeek R1 LLM will analyze the uploaded resume and provide feedback. You can learn about the architecture of the application through the illustration below. Architecture Diagram The React-based user interface communicates with the NodeJS-based backend using REST APIs. NodeJS backend then sends the user request to DeepSeek R1 hosted using Ollama. This entire tech stack can be run on a single machine, as you will do throughout the article, or it can be hosted across multiple containers in more complex use cases. Prerequisites To run the project, you will need a machine with some compute power, preferably one that has an NVIDIA graphics card. The project has been developed and tested on NVIDIA 4090RTX based Windows machine and M2 MacBook Pro. You will need to have NodeJS installed on the machine. This project has been built on NodeJS version 22.3.0. You can verify NodeJS installation using the node -v command. You will also need an editor of your choice to work through the code. Visual Studio Code has been used while building the application and is generally recommended. Setting Up and Running DeepSeek Locally To run DeepSeek R1 locally, follow the steps below: 1. Install Ollama from its official website. 2. After installation is complete, you will be able to run models using the ollama run command from your machine's terminal. 3. Run the DeepSeek model of your choice. This tutorial was built using the DeepSeek R1 8-Billon parameter model. You can run it by using the command ollama run deepseek-r1:8b. 4. If you have a lower-specification machine than the one mentioned in the prerequisites section, the 7B and 1.5B parameter models will work as well, but the generated output quality may be lower. 5. It may take some time for models to run the first time as they will need to get downloaded. Once the model is running, you should be able to ask it a question right in the terminal and get an output. You can refer to the illustration below to view the DeepSeek R1 8B model in action. Ollama DeepSeek R1 6. DeepSeek R1 is a reasoning model, and therefore, it thinks before giving the first answer it can generate. As highlighted in the illustration above, it is thinking before giving the answer to our prompt. This thinking can be seen in tags <think> </think>. Cloning and Running NodeJS Backend The Ollama service can be accessed via an API as well. You are going to leverage this API and build a NodeJS-based backend layer. This layer will take the uploaded PDF from the user and extract text from it. After the text extraction, the backend will feed the text to the DeepSeek R1 model via the Ollama API and get a response back. This response will be sent to the client to display to the user. 1. Clone the backend project from GitHub using this URL. Ideally, you should fork the project and then clone your own local copy. 2. After cloning, to run the project, go to the project root directory using cd deepseek-ollama-backend. 3. Once inside the project root, install dependencies by giving npm install command. Once the installation completes, the project can be run using the npm start command. The core of the project is the app.js file. Examine its code, which is provided below. JavaScript const express = require('express'); const multer = require('multer'); const pdfParse = require('pdf-parse'); const axios = require('axios'); const fs = require('fs'); const cors = require('cors'); const app = express(); app.use(cors()); app.use(express.json()); const upload = multer({ dest: 'uploads/', fileFilter: (req, file, cb) => { file.mimetype === 'application/pdf' ? cb(null, true) : cb(new Error('Only PDF files are allowed!')); } }).single('pdfFile'); app.post('/analyze-pdf', (req, res) => { upload(req, res, async function(err) { if (err) { return res.status(400).json({ error: 'Upload error', details: err.message }); } try { if (!req.file) { return res.status(400).json({ error: 'No PDF file uploaded' }); } const dataBuffer = fs.readFileSync(req.file.path); const data = await pdfParse(dataBuffer); const pdfText = data.text; fs.unlinkSync(req.file.path); const response = await axios.post('http://127.0.0.1:11434/api/generate', { model: "deepseek-r1:8b", prompt: `Analyze this resume. Resume text is between two --- given ahead: ---${pdfText}---`, stream: false }); res.json({ success: true, message: 'Successfully connected to Ollama', ollamaResponse: response.data }); } catch (error) { if (req.file && fs.existsSync(req.file.path)) { fs.unlinkSync(req.file.path); } res.status(500).json({ error: 'Error processing PDF', details: error.message }); } }); }); if (!fs.existsSync('uploads')) { fs.mkdirSync('uploads'); } const PORT = process.env.PORT || 3000; app.listen(PORT, () => { console.log(`Server is running on port ${PORT}`); }); 4. The client interacts with the backend by invoking /analyze-pdf API endpoint, which is of type POST. The client sends the user-uploaded pdf file as a payload to this API. 5. The server stores this file in uploads directory temporarily, and extracts the text in the file. 6. The server then prompts DeepSeek R1 using Ollama's localhost API endpoint. 7. DeepSeek R1 analyzes the content of the resume and provides its feedback. The server then responds to the client with this analysis using res.json(). Cloning and Running the React User Interface The user interface of the project will allow users to upload the resume, send this resume to the backend and then display the result of DeepSeek R1's analysis of the resume to the user. It will also display an internal chain of thoughts or thinking of DeepSeek R1 as well. 1. To get started, fork and then clone the project from this GitHub URL. You can simply clone it as well if you don't intend to do many customizations. 2. Once the project is cloned, go to the root project directory using the command cd deepseek-ollama-frontend. 3. Inside the project root, install the necessary dependencies using the npm install command. After the installation completes, start the project using the npm run dev command. 4. The main component of this React application is ResumeAnalyzer. Open it in your editor of choice and analyze it. JSX import './ResumeAnalyzer.css'; import React, { useState } from 'react'; import { Upload, Loader2 } from 'lucide-react'; import AnalysisSection from './AnalysisSection'; const ResumeAnalyzer = () => { const [file, setFile] = useState(null); const [loading, setLoading] = useState(false); const [feedback, setFeedback] = useState(null); const [error, setError] = useState(null); const handleFileChange = (event) => { const selectedFile = event.target.files?.[0]; if (selectedFile && selectedFile.type === 'application/pdf') { setFile(selectedFile); setError(null); } else { setError('Please upload a PDF file'); setFile(null); } }; const analyzePDF = async () => { if (!file) return; setLoading(true); setError(null); try { const formData = new FormData(); formData.append('pdfFile', file); const response = await fetch('http://localhost:3000/analyze-pdf', { method: 'POST', body: formData, }); if (!response.ok) { const errorData = await response.json(); throw new Error(errorData.details || 'Failed to analyze PDF'); } const data = await response.json(); setFeedback(data); } catch (err) { setError(err.message || 'An error occurred'); } finally { setLoading(false); } }; return ( <div className="max-w-4xl mx-auto p-4"> <div className="bg-gray-50 rounded-lg shadow-lg p-6"> <h1 className="text-3xl font-bold mb-6 text-gray-800">Resume Analyzer</h1> <div className="bg-white rounded-lg shadow-sm p-8"> <div className="border-2 border-dashed border-gray-300 rounded-lg p-8 text-center"> <Upload className="w-12 h-12 text-gray-400 mx-auto mb-4" /> <input type="file" accept=".pdf" onChange={handleFileChange} className="hidden" id="file-upload" /> <label htmlFor="file-upload" className="cursor-pointer text-blue-600 hover:text-blue-800 font-medium" > Upload Resume (PDF) </label> {file && ( <p className="mt-2 text-sm text-gray-600"> Selected: {file.name} </p> )} </div> </div> {error && ( <div className="mt-4 p-4 bg-red-50 text-red-700 rounded-lg border border-red-200"> {error} </div> )} <button onClick={analyzePDF} disabled={!file || loading} className="mt-6 w-full bg-blue-600 text-white py-3 px-4 rounded-lg hover:bg-blue-700 disabled:opacity-50 disabled:cursor-not-allowed flex items-center justify-center font-medium transition-colors" > {loading ? ( <> <Loader2 className="mr-2 h-5 w-5 animate-spin" /> Analyzing Resume... </> ) : ( 'Analyze Resume' )} </button> {feedback && !loading && ( <div className="mt-8"> <h2 className="text-2xl font-bold mb-6 text-gray-800">Analysis Results</h2> {feedback.ollamaResponse && <AnalysisSection ollamaResponse={feedback.ollamaResponse} /> } </div> )} </div> </div> ); }; export default ResumeAnalyzer; 5. This component provides an input field for the user to upload the file. 6. The uploaded file is sent to the server using the API endpoint. 7. The response of the server is divided into two parts- Internal thinking of the model and actual response of the model. 8. The AnalysisSection component displays the response of the model as well as houses the ExpandableSection component, which is used for displaying DeepSeek R1's internal thinking. 9. Navigate to the URL in your browser to load the application. Upload your resume (or any sample resume) and observe the analysis received by DeepSeek R1. Resume Analyzer Conclusion DeepSeek R1 provides a unique opportunity to build GenAI-powered applications completely in-house, and customize them as per your needs. In this article, you have learned about the benefits of using open-source GenAI models. Furthermore, you have set up a real application using DeepSeek R1, Node.js, and React. This setup allows you to perform resume analysis using AI completely offline. You can use this tool to hire smart at your organization, and I advise you to continue building on the knowledge gained from this article and explore more use cases and applications.

By Anujkumarsinh Donvir CORE

Retrieval-augmented generation (RAG) is the most popular approach for obtaining real-time data or updated data from a data source based on text input by users. Thus empowering all our search applications with state-of-the-art neural search. In RAG search systems, each user request is converted into a vector representation by embedding model, and this vector comparison is performed using various algorithms such as cosine similarity, longest common sub-sequence, etc., with existing vector representations stored in our vector-supporting database. The existing vectors stored in the vector database are also generated or updated asynchronously by a separate background process. This diagram provides a conceptual overview of vector comparison To use RAG, we need at least an embedding model and a vector storage database to be used by the application. Contributions from community and open-source projects provide us with an amazing set of tools that help us build effective and efficient RAG applications. In this article, we will implement the usage of a vector database and embedding generation model in a Python application. If you are reading this concept for the first time or nth time, you only need tools to work, and no subscription is needed for any tool. You can simply download tools and get started. Our tech stack consists of the following open-source and free-to-use tools: Operating system – Ubuntu LinuxVector database – Apache CassandraEmbedding model – nomic-embed-textProgramming language – Python Key Benefits of this Stack Open-sourceIsolated data to meet data compliance standards This diagram provides a high-level dependency architecture of the system Implementation Walkthrough You may implement and follow along if prerequisites are fulfilled; otherwise, read to the end to understand the concepts. Prerequisites Linux (In my case, it's Ubuntu 24.04.1 LTS)Java Setup (OpenJDK 17.0.2)Python (3.11.11)Ollama Ollama Model Setup Ollama is an open-source middleware server that acts as an abstraction between generative AI and applications by installing all the necessary tools to make generative AI models available to consume as CLI and API in a machine. It has most of the openly available models like llama, phi, mistral, snowflake-arctic-embed, etc. It is cross-platform and can be easily configured in OS. In Ollama, we will pull the nomic-embed-text model to generate embeddings. Run in command line: Plain Text ollama pull nomic-embed-text This model generates embeddings of size 768 vectors. Apache Cassandra Setup and Scripts Cassandra is an open-source NoSQL database designed to work with a high amount of workloads that require high scaling as per industry needs. Recently, it has added support for Vector search in version 5.0 that will facilitate our RAG use case. Note: Cassandra requires Linux OS to work; it can also be installed as a docker image. Installation Download Apache Cassandra from https://cassandra.apache.org/_/download.html. Configure Cassandra in your PATH. Start the server by running the following command in the command line: Plain Text cassandra Table Open a new Linux terminal and write cqlsh; this will open the shell for Cassandra Query Language. Now, execute the below scripts to create the embeddings keyspace, document_vectors table, and necessary index edv_ann_index to perform a vector search. SQL CREATE KEYSPACE IF NOT EXISTS embeddings WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : '1' }; USE embeddings; CREATE TABLE IF NOT EXISTS embeddings.document_vectors ( record_id timeuuid, id uuid, content_chunk text, content_vector VECTOR <FLOAT, 768>, created_at timestamp, PRIMARY KEY (id, created_at) ) WITH CLUSTERING ORDER BY (created_at DESC); CREATE INDEX IF NOT EXISTS edv_ann_index ON embeddings.document_vectors(content_vector) USING 'sai'; Note: content_vector VECTOR <FLOAT, 768> is responsible for storing vectors of 768 length that are generated by the model. Milestone 1: We are ready with database setup to store vectors. Python Code This programming language certainly needs no introduction; it is easy to use and loved by the industry with strong community support. Virtual Environment Set up virtual environment: Plain Text sudo apt install python3-virtualenv && python3 -m venv myvenv Activate virtual environment: Plain Text source /media/setia/Data/Tutorials/cassandra-ollama-app/myvenv/bin/activate Packages Download Datastax Cassandra package: Plain Text pip install cassandra-driver Download requests package: Plain Text pip install requests File Create a file named app.py. Now, write the code below to insert sample documents in Cassandra. This is the first step always to insert data in the database; it can be done by a separate process asynchronously. For demo purposes, I have written a method that will insert documents first in the database. Later on, we can comment on this method once the insertion of documents is successful. Python from cassandra.cluster import Cluster from cassandra.query import PreparedStatement, BoundStatement import uuid import datetime import requests cluster = Cluster(['127.0.0.1'],port=9042) session = cluster.connect() def generate_embedding(text): embedding_url = 'http://localhost:11434/api/embed' body = { "model": "nomic-embed-text", "input": text } response = requests.post(embedding_url, json = body) return response.json()['embeddings'][0] def insert_chunk(content, vector): id = uuid.uuid4() content_chunk = content content_vector = vector created_at = datetime.datetime.now() insert_query = """ INSERT INTO embeddings.document_vectors (record_id, id, content_chunk, content_vector, created_at) VALUES (now(), ?, ?, ?, ?) """ prepared_stmt = session.prepare(insert_query) session.execute(prepared_stmt, [ id, content_chunk, content_vector, created_at ]) def insert_sample_data_in_cassandra(): sentences = [ "The aroma of freshly baked bread wafted through the quaint bakery nestled in the cobblestone streets of Paris, making Varun feel like time stood still.", "Sipping a spicy masala chai in a bustling tea stall in Mumbai, Varun felt he was tasting the very soul of the city.", "The sushi in a small Tokyo diner was so fresh, it felt like Varun was on a culinary journey to the sea itself.", "Under the starry desert sky in Morocco, Varun enjoyed a lamb tagine that tasted like a dream cooked slowly over a fire.", "The cozy Italian trattoria served the creamiest risotto, perfectly capturing the heart of Tuscany on a plate, which Varun savored with delight.", "Enjoying fish tacos on a sunny beach in Mexico, with the waves crashing nearby, made the flavors unforgettable for Varun.", "The crispy waffles drizzled with syrup at a Belgian café were worth every minute of waiting, as Varun indulged in the decadent treat.", "A bowl of warm pho in a roadside eatery in Hanoi felt like comfort wrapped in a broth of herbs and spices, giving Varun a sense of warmth.", "Sampling chocolate truffles in a Swiss chocolate shop, Varun found himself in a moment of pure bliss amidst snow-capped mountains.", "The street food stalls in Bangkok served fiery pad Thai that left Varun with a tangy memory of the city’s vibrant energy." ] for sentence in sentences: vector = generate_embedding(sentence) insert_chunk(sentence, vector) insert_sample_data_in_cassandra() Now, run this file using the commandline in the virtual environment: Plain Text python app.py Once the file is executed and documents are inserted, this can be verified by querying the Cassandra database from the cqlsh console. For this, open cqlsh and execute: SQL SELECT content_chunk FROM embeddings.document_vectors; This will return 10 documents inserted in the database, as seen in the screenshot below. Milestone 2: We are done with data setup in our vector database. Now, we will write code to query documents based on cosine similarity. Cosine similarity is the dot product of two vector values. Its formula is A.B / |A||B|. This cosine similarity is internally supported by Apache Cassandra, helping us to compute everything in the database and handle large data efficiently. The code below is self-explanatory; it fetches the top three results based on cosine similarity using ORDER BY <column name> ANN OF <text_vector> and also returns cosine similarity values. To execute this code, we need to ensure that indexing is applied to this vector column. Python def query_rag(text): text_embeddings = generate_embedding(text) select_query = """ SELECT content_chunk,similarity_cosine(content_vector, ?) FROM embeddings.document_vectors ORDER BY content_vector ANN OF ? LIMIT 3 """ prepared_stmt = session.prepare(select_query) result_rows = session.execute(prepared_stmt, [ text_embeddings, text_embeddings ]) for row in result_rows: print(row[0], row[1]) query_rag('Tell about my Bangkok experiences') Remember to comment insertion code: Python #insert_sample_data_in_cassandra() Now, execute the Python code by using python app.py. We will get the output below: Plain Text (myvenv) setia@setia-Lenovo-IdeaPad-S340-15IIL:/media/setia/Data/Tutorials/cassandra-ollama-app$ python app.py The street food stalls in Bangkok served fiery pad Thai that left Varun with a tangy memory of the city’s vibrant energy. 0.8205469250679016 Sipping a spicy masala chai in a bustling tea stall in Mumbai, Varun felt he was tasting the very soul of the city. 0.7719690799713135 A bowl of warm pho in a roadside eatery in Hanoi felt like comfort wrapped in a broth of herbs and spices, giving Varun a sense of warmth. 0.7495554089546204 You can see the cosine similarity of "The street food stalls in Bangkok served fiery pad Thai that left Varun with a tangy memory of the city’s vibrant energy." is 0.8205469250679016, which is the closest match. Final Milestone: We have implemented the RAG search. Enterprise Applications Apache Cassandra For enterprises, we can use Apache Cassandra 5.0 from popular cloud vendors such as Microsoft Azure, AWS, GCP, etc. Ollama This middleware requires a VM compatible with Nvidia-powered GPU for running high-performance models, but we don't need high-end VMs for models used for generating vectors. Depending upon traffic requirements, multiple VMs can be used, or any generative AI service like Open AI, Anthropy, etc, whichever Total Cost of Ownership is lower for scaling needs or Data Governance needs. Linux VM Apache Cassandra and Ollama can be combined and hosted in a single Linux VM if the use case doesn't require high usage to lower the Total Cost of Ownership or to address Data Governance needs. Conclusion We can easily build RAG applications by using Linux OS, Apache Cassandra, embedding models (nomic-embed-text) used via Ollama, and Python with good performance without needing any additional cloud subscription or services in the comfort of our machines/servers. However, hosting a VM in server(s) or opt for a cloud subscription for scaling as an enterprise application compliant with scalable architectures is recommended. In this Apache, Cassandra is a key component to do the heavy lifting of our vector storage and vector comparison and Ollama server for generating vector embeddings. That's it! Thanks for reading 'til the end.

Distributed systems have been there for a while now and there are well-known patterns already established when designing them. Today, we will discuss one of the popular patterns: "locks." Simply put, locks are how processes gain exclusive access to a resource to perform a certain action. For example, imagine there are a bunch of Blobs in a storage account, and you need one instance of your service to process each blob to avoid duplicate processing. The way to do it would be to acquire a lock on the blob, complete processing, and release it. However, a potential issue arises if a process fails before releasing the lock, either because the process died or due to a network partition, leaving the resource locked indefinitely. This can lead to deadlocks and resource contention. To prevent deadlocks, one strategy that can be employed is to use timeouts or leased-based locks. Timeout Lock In this case, there is a predefined timeout the process requests the lock for. If the lock is not released before the timeout, the system ensures the lock is eventually released. Lease Lock For lease-based locks, a renew lease API is provided alongside the timeout mechanism. The process holding the lock must call this API before the lease expires to maintain exclusive access to the resource. If the process fails to renew the lease in time, the lock is automatically released, allowing other processes to acquire it. Pros and Cons of Timeout and Lease-Based Locks ProsConsTimeout based lockSimple to implementRequires careful selection of the timeoutPrevent permanent locksIf the processing is not complete, then there is no way to renew the leaseLease based lockReduces risk of premature lock expirationRequires mechanism for lease renewalProcess can continue to request the lease until work is complete. Both the above strategies are a way to quickly recover from process failures or network partitions in distributed systems. Lease Lock Strategy With Azure Storage Let's look at how to use the Lease Lock strategy with Azure Storage. This also covers the Timeout lock strategy. Step 1: Import the Storage Blob Nuget "12.23.0" is the latest version at the time of authoring this article. The latest versions can be found at Azure Storage Blobs. XML <ItemGroup> <PackageReference Include="Azure.Storage.Blobs" Version="12.23.0" /> </ItemGroup> Step 2: Acquire the Lease Below is the code to acquire the lease. C# public async Task<string> TryAcquireLeaseAsync(string blobName, TimeSpan durationInSeconds, string leaseId = default) { BlobContainerClient blobContainerClient = new BlobContainerClient(new Uri($"https://{storageName}.blob.core.windows.net/processors"), tokenCredential, blobClientOptions); BlobLeaseClient blobLeaseClient = blobContainerClient.GetBlobClient(blobName).GetBlobLeaseClient(leaseId); try { BlobLease lease = await blobLeaseClient.AcquireAsync(durationInSeconds).ConfigureAwait(false); return lease.LeaseId; } catch (RequestFailedException ex) when (ex.Status == 409) { return default; } } First, we create a Blob Container Client and retrieve the Blob Client for the specific blob we want to acquire a lease on.Second, the "Acquire Async" method tries to acquire the lease for a specific duration. If the acquisition was successful, a lease Id is returned if not a 409 (Status code for conflict) is thrown.The "Acquire Async" is the key method here. The rest of the code can be tailored/edited as per your needs. Step 3: Renew the Lease "Renew Async" is the method in the Storage .NET SDK used for renewing the lease.If the renewal is unsuccessful an exception is thrown along with the reason for the cause of failure. C# public async Task ReleaseLeaseAsync(string blobName, string leaseId) { BlobLeaseClient blobLeaseClient = this.blobContainerClient.GetBlobClient(blobName).GetBlobLeaseClient(leaseId); await blobLeaseClient.RenewAsync().ConfigureAwait(false); } Step 4: Orchestrate the Acquire and Renew Lease Methods Initially, we call the "Try Acquire Lease Async" to fetch the lease identifier from Step 2. Once it is successful, a background task is kicked off that calls the "Renew Lease Async" from Step 3 every X seconds. Just make sure there is enough time between the timeout and when the renew lease method is called. C# string leaseId = await this.blobReadProcessor.TryAcquireLeaseAsync(blobName, TimeSpan.FromSeconds(60)).ConfigureAwait(false); Task leaseRenwerTask = this.taskFactory.StartNew( async () => { while (leaseId != default && !cancellationToken.IsCancellationRequested) { await Task.Delay(renewLeaseMillis).ConfigureAwait(false); await this.blobReadProcessor.RenewLeaseAsync(blobName, leaseId).ConfigureAwait(false); } }, CancellationToken.None, TaskCreationOptions.LongRunning, TaskScheduler.Default); The cancellation token is used to gracefully stop the lease renewal task when it's no longer needed. Step 5: Cancel the Lease Renewal When the "Cancel Async" method is called, the "IsCancellationRequested" in Step 4 becomes true, because of which we no longer enter the while loop and request for lease renewal. C# await cancellationTokenSource.CancelAsync().ConfigureAwait(false); await leaseRenwerTask.WaitAsync(Timeout.InfiniteTimeSpan).ConfigureAwait(false); Step 6: Release the Lease Finally, to release the lease just call the "Release Async" method. C# public async Task ReleaseLeaseAsync(string blobName, string leaseId) { BlobLeaseClient blobLeaseClient = this.blobContainerClient.GetBlobClient(blobName).GetBlobLeaseClient(leaseId); await blobLeaseClient.ReleaseAsync().ConfigureAwait(false); } Conclusion Locks are among the fundamental patterns in distributed systems to gain exclusive access to resources. It is necessary to keep the pitfalls in mind while dealing with them for the smooth running of operations. By using Azure Storage, we can implement these efficient locking mechanisms that can prevent indefinite blocking and, at the same time, provide elasticity in how the locks are maintained.

By Siri Varma Vegiraju CORE