Coding

In today’s digital era, cybersecurity is a cornerstone of maintaining trust and reliability in cloud operations. A managed threat detection service by AWS, like Amazon GuardDuty, can help secure your environment by analyzing activity and identifying potential risks. This hands-on guide will help you enable Amazon GuardDuty on your AWS account and begin monitoring your resources for security threats. Amazon GuardDuty is a threat detection service that uses machine learning, anomaly detection, and integrated threat intelligence to protect your AWS environment. It continuously monitors for malicious activity, unauthorized access, and security vulnerabilities by analyzing data sources like AWS CloudTrail logs, VPC Flow Logs, and DNS logs. Benefits of GuardDuty Automated threat detection: GuardDuty identifies suspicious behavior in real time, such as unusual API calls, unauthorized access attempts, and data exfiltration activities.Ease of use: There’s no need to deploy or manage additional security infrastructure — GuardDuty is fully managed by AWS.Cost-effective: You only pay for what you use, making it an affordable solution for proactive threat detection.Seamless integration: GuardDuty integrates with other AWS security tools such as AWS Security Hub, Amazon CloudWatch, and Amazon SNS for notifications. How to Enable Amazon GuardDuty Follow these steps to enable GuardDuty on your AWS account: Step 1: Prepare Your AWS Account Before you begin, ensure that: You have an active AWS account.Your IAM user or role has the necessary permissions. Assign the AmazonGuardDutyFullAccess policy to the user or role to enable and manage GuardDuty. Step 2: Access GuardDuty in the AWS Console Sign in to the AWS Management Console.Navigate to the GuardDuty service under the Security, Identity, and Compliance section. Step 3: Enable the Service On the GuardDuty dashboard, click Get Started or Enable GuardDuty.Review the terms of use and configurations.Confirm the setup by clicking Enable. Once GuardDuty is activated, it will begin analyzing data from various sources like CloudTrail logs, VPC Flow Logs, and DNS queries to detect anomalies. Note: You can choose one of the options below to enable Guard Duty: Try threat detection with GuardDutyGuardDuty Malware Protection for S3 only Step 4: Configure Multi-Account Support (Optional) If you manage multiple AWS accounts, consider enabling multi-account support. Use AWS Organizations to designate a GuardDuty administrator account that can manage the service across all linked accounts. Step 5: Monitor and Respond to Findings After enabling GuardDuty, its findings will populate the dashboard. GuardDuty classifies findings by severity — low, medium, or high — allowing you to prioritize actions. Integrate GuardDuty with: AWS Security Hub: For centralized security management.Amazon CloudWatch: To set up alarms and trigger workflows.Amazon SNS: For email or SMS notifications about threats. Best Practices for Using GuardDuty Enable logging: Ensure that CloudTrail logs and VPC Flow Logs are active for comprehensive monitoring.Integrate with automation: Use AWS Lambda to automate responses to high-severity findings.Review regularly: Periodically review findings and update security policies based on GuardDuty insights. Conclusion Amazon GuardDuty is a helpful tool for improving the security of your AWS environment. Enabling this service will help you stay proactive in detecting and responding to potential threats. Its ease of use and robust threat detection capabilities make it a valuable option for organizations using AWS. Author's Note: Take the first step today by enabling GuardDuty on your AWS account to protect your cloud environment against modern security challenges.

By Sai Sandeep Ogety CORE

Think back to those days when you met the love of your life. The feeling was mutual. The world seemed like a better place, and you were on an exciting journey with your significant other. You were both “all-in” as you made plans for a life together. Life was amazing... until it wasn’t. When things don’t work out as planned, then you’ve got to do the hard work of unwinding the relationship. Communicating with each other and with others. Sorting out shared purchases. Moving on. Bleh. Believe it or not, our relationship with technology isn’t all that different. Breaking Up With a Service There was a time when you decided to adopt a service — maybe it was a SaaS, or a PaaS, or something more generic. Back in the day, did you make the decision while also considering the time when you would no longer plan to use the service anymore? Probably not. You were just thinking of all the wonderful possibilities for the future. But what happens when going with that service is no longer in your best interest? Now, you’re in for a challenge, and it’s called service abdication. While services can be shut down with a reasonable amount of effort, getting the underlying data can be problematic. This often depends on the kind of service and the volume of data owned by that service provider. Sometimes, the ideal unwinding looks like this: Stop paying for the service, but retain access to the data source for some period of time. Is this even a possibility? Yes, it is! The Power of VPC Peering Leading cloud providers have embraced the virtual private cloud (VPC) network as the de facto approach to establishing connectivity between resources. For example, an EC2 instance on AWS can access a data source using VPCs and VPC end-point services. Think of it as a point-to-point connection. VPCs allow us to grant access to other resources in the same cloud provider, but we can also use them to grant access to external services. Consider a service that was recently abdicated but with the original data source left in place. Here’s how it might look: This concept is called VPC peering, and it allows for a private connection to be established from another network. A Service Migration Example Let’s consider a more concrete example. In your organization, a business decision was made to streamline how it operates in the cloud. While continuing to leverage some AWS services, your organization wanted to optimize how it builds, deploys, and manages its applications by terminating a third-party, cloud-based service running on AWS. They ran the numbers and concluded that internal software engineers could stand up and support a new auto-scaled service on Heroku for a fraction of the cost that they had been paying the third-party provider. However, because of a long tenure with the service provider, migrating the data source is not an option anytime soon. You don’t want the service, and you can’t move the data, but you still want access to the data. Fortunately, the provider has agreed to a new contract to continue hosting the data and provide access — via VPC peering. Here’s how the new arrangement would look: VPC Peering With Heroku In order for your new service (a Heroku app) to access the original data source in AWS, you’ll first need to run your app within a Private Space. For more information, you can read about secure cloud adoption and my discovery of Heroku Private Spaces. Next, you’ll need to meet the following simple network requirements: The VPC must use a compatible IPv4 CIDR block in its network configuration.The VPC must use an RFC1918 CIDR block (10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16).The VPC’s CIDR block must not overlap with the CIDR ranges for your Private Space. The default ranges are 10.0.0.0/16, 10.1.0.0/16, and 172.17.0.0/16. With your Private Space up and running, you’ll need to retrieve its peering information: Shell $ heroku spaces:peering:info our-new-app === our-new-app Peering Info AWS Account ID: 647xxxxxx317 AWS Region: us-east-1 AWS VPC ID: vpc-e285ab73 AWS VPC CIDR: 10.0.0.0/16 Space CIDRs: 10.0.128.0/20, 10.0.144.0/20, 10.0.0.0/20, 10.0.16.0/20 Unavailable CIDRs: 10.1.0.0/16 Copy down the AWS Account ID (647xxxxxx317) and AWS VPC ID (vpc-e285ab73). You’ll need to give that information to the third-party provider who controls the AWS data source. From there, they can use either the AWS Console or CLI to create a peering connection. Their operation would look something like this: Shell $ aws ec2 create-vpc-peering-connection \ --vpc-id vpc-e527bb17 \ --peer-vpc-id vpc-e285ab73 \ --peer-owner-id 647xxxxxx317 { "VpcPeeringConnection": { "Status": { "Message": "Initiating Request to 647xxxxxx317", "Code": "initiating-request" }, "Tags": [], "RequesterVpcInfo": { "OwnerId": "714xxxxxx214", "VpcId": "vpc-e527bb17", "CidrBlock": "10.100.0.0/16" }, "VpcPeeringConnectionId": "pcx-123abc456", "ExpirationTime": "2025-04-23T22:05:27.000Z", "AccepterVpcInfo": { "OwnerId": "647xxxxxx317", "VpcId": "vpc-e285ab73" } } } This creates a request to peer. Once the provider has done this, you can view the pending request on the Heroku side: Shell $ heroku spaces:peerings our-new-app In the screenshot below, we can see the pending-acceptance status for the peering connection. From here, you can accept the peering connection request: Shell $ heroku spaces:peerings:accept pcx-123abc456 --space our-new-app Accepting and configuring peering connection pcx-123abc456 We check the request status a second time: Shell $ heroku spaces:peerings our-new-app We see that the peer connection is active. At this point, the app running in our Heroku Private Space will be able to access the AWS data source without any issues. Conclusion An unfortunate truth in life is that relationships can be unsuccessful just as often as they can be long-lasting. This applies to people, and it applies to technology. When it comes to technology decisions, sometimes changing situations and needs drive us to move in different directions. Sometimes, things just don’t work out. And in these situations, the biggest challenge is often unwinding an existing implementation — without losing access to persistent data. Fortunately, Heroku provides a solution for slowly migrating away from existing cloud-based solutions while retaining access to externally hosted data. Its easy integration for VPC peering with AWS lets you access resources that still need to live in the legacy implementation, even if the rest of you have moved on. Taking this approach will allow your new service to thrive without an interruption in service to the consumer.

By John Vester CORE

Recently, I haven’t been updating my open-source articles as frequently — not because I’ve stopped writing, but because the progress on open-source commercialization has been great, and the endless task of drafting proposals has consumed my time. As a native open-source commercial company, WhaleOps employs mostly engineers. Asking these open-source contributors to write proposals wastes their development time, and their proposals don’t always meet the quality standard. Unlike managing in a big company, being a startup CEO means stepping into any role the company needs. After every strategic meeting, I’m the first to roll up my sleeves and tackle the most urgent tasks. As a result, I haven’t had time to write articles, as my limited time is mostly taken up with creating proposals that follow the formal template style. Especially recently, with one bid after another, I’ve found myself questioning my own sanity. The existing large models coudn't solve my problem, so I built my own. As a tech person, I always look for tools to solve my problems. Naturally, I thought of large models, but they can’t fully grasp the nuances of our products and often produce unreliable outputs. Plus, you’re the one delivering the work, not the model. So, I decided to develop a tool for proposal generation based on large models using Python and a compatible large model (compatible with ChatGPT). This tool automatically generates a proposal from your product documentation, breaking down product manuals into feature points. Based on a human-created mapping of these points to the requirements, it generates a Word version of the proposal and an Excel deviation table. The model can condense or expand content or simply copy relevant functionality as needed. Features of the Open-Source Proposal Tool The functionality of this tool is simple, with the most challenging part being the Word formatting (formatting in Word is always a pain). I experimented with several methods to make sure it follows the correct Title 1, Title 2, Title 3, body text, table, and image formats in the Word template. Staying true to the open-source spirit, I’ve uploaded the tool to my personal GitHub under an Apache License, so feel free to use it if you need it. Here’s what it does: Breaks down your product manual into a set of reusable detail documents, reducing the need to repeatedly reference the source document when drafting a proposal. You can also customize specific functionality (the default file name is “Template.docx”).Based on the requirements table filled in by a person, it automatically generates a proposal in a point-to-point response format, including all headings and content, with the correct Title 1, 2, 3 formatting, and automatically organizes body text, images, and bullet points (the default requirements table is “requirements_table.xlsx,” and the generated content is in “proposal_content.docx”).For any product requirements in the corresponding functionality section, it automatically copies the product manual content into the point-to-point response section, retaining images, tables, and bullet points. You can also rewrite the product description to suit different proposal needs. If there’s no matching functionality, the model automatically generates relevant content (review and modify as needed).Completes the technical requirements deviation table by automatically filling in responses in the “requirements_table.xlsx,” with responses formatted as “Answer: Fully supports, {Model-generated text based on project requirements}” and includes the section number corresponding to the proposal. With this tool, you can quickly modify and generate proposals at will. You can get it here. Proposal Generation Process Start by running Extract_Word.py to generate your product feature points as a Word document, then run Generate.py. If a feature point is missing, you can enter “X” in the Excel file, and the model will generate content that meets the requirements. However, I strongly recommend a manual review. After generation, you’ll see the proposal formatted with a table of contents, body text, images, tables, and bullet points, all automatically organized. The entire proposal is structured according to the client’s requirements in sequential format, with subheadings, content, images, and point-to-point responses. This takes care of all the repetitive work. The automatically generated deviation table includes everything, whether the content was generated by the model or not, along with the corresponding section numbers. You only need to finalize the deviation table with “&” symbols based on the final requirements — the model has written it all for you. For me, this tool has reduced what used to be 8 hours of work to around 30 minutes, and even our business team can generate the business proposal sections using the template. In total, this has cut down our time for a week-long proposal to 1-2 days, reducing the manpower required by 80%. How to Use It? First, download all the code to a single directory from GitHub: Proposal Large Model (Chinese version). Install the Python environment and packages: pip install openpyxl, docx, openai, requests, docx, python-docx.Apply for a ChatGPT or Baidu Qianfan large model key (I used ERNIE-Speed-8K, which is free), record the token, and place it in the relevant code section.Copy your product manual to Template.docx. Be sure to use the body text, Title 1, Title 2, and Title 3 styles provided; other formats may cause issues.Run Extract_Word.py to generate the feature point document from the product manual (supports up to 3 heading levels). If the list formatting appears off, don’t worry; the final format will align properly.Fill in Columns B and C (which will generate secondary and tertiary headings automatically) and Column G (the corresponding product manual chapter). If a chapter is missing, enter “X.” Note that if there is no corresponding chapter or an “X” is entered, the model will generate content automatically.Review the “proposal_content.docx” document and keep the chapter for which you want to start generating the proposal. You can modify the body text and heading 1, 2, and 3 styles; just don’t rename the styles, or there may be errors.Adjust the parameters in Generate.py: API_KEY and SECRET_KEY: Baidu Cloud large model keys.MAX_WIDTH_CM: Maximum image width; images larger than this will be resized automatically.The prompts for generating content have been customized for large data scenarios, but feel free to modify them.MoreSection=1 will read Column C to generate detailed tertiary headings (default is on).ReGenerateText=0 will re-generate text content automatically for different proposal needs (default is off).DDDAnswer=1 generates the point-to-point response content at the top of each feature point (default is on).key_flag=1 adds the importance level of each requirement to the proposal headings (default is on).last_heading_1=2 specifies the starting chapter for the technical solution in “proposal_content.docx.”Run Generate.py. Summary With this tool, you only need to check the product features against the proposal requirements, and most of the proposal content is generated automatically. The business proposal section can also be generated similarly, so creating a 1,000-page proposal now only takes a few minutes.

As a developer learning Rust, I wanted to build a practical project to apply my new skills. With the rise of large language models like Anthropic's Llama 3.2, I thought it would be interesting to create a Rust command line interface (CLI) to interact with the model. In just a couple of minutes, I was able to put together a working CLI using the Ollama Rust library. The CLI, which I call "Jarvis," allows you to chat with Llama 3.2, as well as perform some basic commands like checking the time, date, and listing directory contents. In this post, I'll walk through the key components of the Jarvis CLI and explain how you can use Rust to interface with Llama 3.2 or other large language models. By the end, you'll see how Rust's performance and expressiveness make it a great choice for AI applications. The Jarvis CLI Structure The main components of the Jarvis CLI include: 1. JarvisConfig Struct Defines the available commandsMethods to validate commands and print help text 2. Command Handling Logic in main() Parses command line argumentsInvokes the appropriate function based on the command 3. Functions for Each Command time - Gets current timedate - Gets today's datehello - Prints a customizable greetingls - Lists directory contentschat - Interacts with Llama 3.2 using Ollama lib Here's a condensed version of the code: Plain Text struct JarvisConfig { commands: Vec<&'static str>, } impl JarvisConfig { fn new() -> Self {...} fn print_help(&self) {...} fn is_valid_command(&self, command: &str) -> bool {...} } #[tokio::main] async fn main() { let config = JarvisConfig::new(); let args: Vec<String> = env::args().collect(); match args[1].as_str() { "time" => {...} "date" => {...} "hello" => {...} "ls" => {...} "chat" => { let ollama = Ollama::default(); match ollama .generate(GenerationRequest::new( "llama3.2".to_string(), args[2].to_string(), )) .await { Ok(res) => println!("{}", res.response), Err(e) => println!("Failed to generate response: {}", e), } } _ => { println!("Unknown command: {}", args[1]); config.print_help(); } } } Using Ollama to Chat with Llama 3.2 The most interesting part is the "chat" command, which interfaces with Llama 3.2 using the Ollama Rust library. After adding the Ollama dependency to Cargo.toml, using it is fairly straightforward: 1. Create an Ollama instance with default settings: Plain Text let ollama = Ollama::default(); 2. Prepare a GenerationRequest with the model name and prompt: Plain Text GenerationRequest::new( "llama3.2".to_string(), args[2].to_string() ) 3. Asynchronously send the request using ollama.generate(): Plain Text match ollama.generate(...).await { Ok(res) => println!("{}", res.response), Err(e) => println!("Failed to generate response: {}", e), } That's it! With just a few lines of code, we can send prompts to Llama 3.2 and receive generated responses. Example Usage Here are some sample interactions with the Jarvis CLI: Plain Text $ jarvis hello Hello, World! $ jarvis hello Alice Hello, Alice! $ jarvis time Current time in format (HH:mm:ss): 14:30:15 $ jarvis ls /documents /documents/report.pdf: file /documents/images: directory $ jarvis chat "What is the capital of France?" Paris is the capital and most populous city of France. While Python remains the go-to for AI/ML, Rust is a compelling alternative where maximum performance, concurrency, and/or safety are needed. It's exciting to see Rust increasingly adopted in this space. Conclusion In this post, we learned how to build a Rust CLI to interact with Llama 3.2 using the Ollama library. With basic Rust knowledge, we could put together a useful AI-powered tool in just a couple of minutes. Rust's unique advantages make it well-suited for AI/ML systems development. As the ecosystem matures, I expect we'll see even more adoption. I encourage you to try out Rust for your next AI project, whether it's a simple CLI like this or a more complex system. The performance, safety, and expressiveness may surprise you.

API testing has gained a lot of momentum these days. As UI is not involved, it is a lot easier and quicker to test. This is the reason why API testing is considered the first choice for performing end-to-end testing of the system. Integrating the automated API Tests with the CI/CD pipelines allows teams to get faster feedback on the builds. In this blog, we'll discuss and learn about DELETE API requests and how to handle them using Playwright Java for automation testing, covering the following points: What is a DELETE request?How do you test DELETE APIs using Playwright Java? Getting Started It is recommended that you check out the earlier tutorial blog to learn about the details related to prerequisites, setup, and configuration. Application Under Test We will be using the free-to-use RESTful e-commerce APIs that offer multiple APIs related to order management functionality, allowing us to create, retrieve, update, and delete orders. This application can be set up locally using Docker or NodeJS. What Is a DELETE Request? A DELETE API request deletes the specified resource from the server. Generally, there is no response body in the DELETE requests. The resource is specified by a URI, and the server permanently deletes it. DELETE requests are neither considered safe nor idempotent, as they may cause side effects on the server, like removing data from a database. The following are some of the limitations of DELETE requests: The data deleted using a DELETE request is not reversible, so it should be handled carefully.It is not considered to be a safe method as it can directly delete the resource from the database, causing conflicts in the system.It is not an idempotent method, meaning calling it multiple times for the same resource may result in different states. For example, in the first instance, when DELETE is called, it will return Status Code 204 stating that the resource has been deleted, and if DELETE is called again on the same resource, it may give a 404 NOT FOUND as the given resource is already deleted. The following is an example of the DELETE API endpoint from the RESTful e-commerce project. DELETE /deleteOrder/{id} :  Deletes an Order By ID This API requires the order_id to be supplied as Path Parameter in order to delete respective order from the system. There is no request body required to be provided in this DELETE API request. However, as a security measure, the token is required to be provided as a header to delete the order. Once the API is executed, it deletes the specified order from the system and returns Status Code 204. In case where the order is not found, or the token is not valid or not provided, it will accordingly show the following response: Status CodeDescription400 Failed to authenticate the token404 No order with the given order_id is found in the system403Token is missing in the request How to Test DELETE APIs Using Playwright Java Testing DELETE APIs is an important step in ensuring the stability and reliability of the application. Correct implementation of the DELETE APIs is essential to check for unintended data loss and inconsistencies, as the DELETE APIs are in charge of removing the resources from the system. In this demonstration of testing DELETE APIs using Playwright Java, we'll be using the /deleteOrder/{id} for deleting an existing order from the system. Test Scenario 1:  Delete a Valid Order Start the RESTful e-commerce service.Using a POST request, create some orders in the system.Delete the order with order_id “1” using DELETE request.Check that the Status Code 204 is returned in the response. Test Implementation The following steps are required to be performed to implement the test scenario: Add new orders using the POST request.Hit the /auth API to generate token.Hit the /deleteOrder/ API endpoint with the token and the order_id to delete the order.Check that the Status Code 204 is returned in the response. A new test method,  testShouldDeleteTheOrder(), is created in the existing test class HappyPathTests. This test method implements the above three steps to test the DELETE API. Java @Test public void testShouldDeleteTheOrder() { final APIResponse authResponse = this.request.post("/auth", RequestOptions.create().setData(getCredentials())); final JSONObject authResponseObject = new JSONObject(authResponse.text()); final String token = authResponseObject.get("token").toString(); final int orderId = 1; final APIResponse response = this.request.delete("/deleteOrder/" + orderId, RequestOptions.create() .setHeader("Authorization", token)); assertEquals(response.status(), 204); } The POST /auth API endpoint will be hit first to generate the token. The token received in response is stored in the token variable to be used further in the DELETE API request. Next, new orders will be generated using the testShouldCreateNewOrders() method, which is already discussed in the previous tutorial, where we talked about testing POST requests using Playwright Java. After the orders are generated, the next step is to hit the DELETE request with the valid order_id that would delete the specific order. We'll be deleting the order with the order_id “1” using the delete() method provided by Playwright framework. After the order is deleted, the Status Code 204 is returned in response. An assertion will be performed on the Status Code to verify that the Delete action was successful. Since no request body is returned in the response, this is the only thing that can be verified. Test Execution We'll be creating a new testng.xml named testng-restfulecommerce-deleteorders.xml to execute the tests in the order of the steps that we discussed in the test implementation. XML <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> <suite name="Restful ECommerce Test Suite"> <test name="Testing Happy Path Scenarios of Creating and Updating Orders"> <classes> <class name="io.github.mfaisalkhatri.api.restfulecommerce.HappyPathTests"> <methods> <include name="testShouldCreateNewOrders"/> <include name="testShouldDeleteTheOrder"/> </methods> </class> </classes> </test> </suite> First, the testShouldCreateNewOrders() test method will be executed, and it will create new orders. Next, the testShouldDeleteTheOrder() test method order will be executed to test the delete order API. The following screenshot of the test execution performed using IntelliJ IDE shows that the tests were executed successfully. Now, let’s verify that the order was correctly deleted by writing a new test that will call the GET /getOrder API endpoint with the deleted order_id. Test Scenario 2:  Retrieve the Deleted Order Delete a valid order with order_id “1.”Using GET /getOrder API, try retrieving the order with order_id “1.”Check that the Status Code 404 is returned with the message “No Order found with the given parameters!” in the response. Test Implementation Let’s create a new test method, testShouldNotRetrieveDeletedOrder(), in the existing class HappyPathTests. Java @Test public void testShouldNotRetrieveDeletedOrder() { final int orderId = 1; final APIResponse response = this.request.get("/getOrder", RequestOptions.create().setQueryParam("id", orderId)); assertEquals(response.status(), 404); final JSONObject jsonObject = new JSONObject(response.text()); assertEquals(jsonObject.get("message"), "No Order found with the given parameters!"); } The test implementation of this scenario is pretty simple. We will be executing the GET /getOrder API and to fetch the deleted order with order_id “1.” An assertion is applied next to verify that the GET API should return the Status Code 404 in the response with the message “No Order found with the given parameters!” This test ensures that the delete order API worked fine and the order was deleted from the system. Test Execution Let’s update the testng.xml file and add this test scenario at the end after the delete test. Java <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> <suite name="Restful ECommerce Test Suite"> <test name="Testing Happy Path Scenarios of Creating and Updating Orders"> <classes> <class name="io.github.mfaisalkhatri.api.restfulecommerce.HappyPathTests"> <methods> <include name="testShouldCreateNewOrders"/> <include name="testShouldDeleteTheOrder"/> <include name="testShouldNotRetrieveDeletedOrder"/> </methods> </class> </classes> </test> </suite> Now, all three tests should run in sequence. The first one will create orders; the second one will delete the order with order_id “1”; and the last test will hit the GET API to fetch the order with order_id “1” returning Status Code 404. The screenshot above shows that all three tests were executed successfully, and the DELETE API worked fine as expected. Summary DELETE API requests allow the deletion of the resource from the system. As delete is an important CRUD function, it is important to test it and verify that the system is working as expected. However, it should be noted that DELETE is an irreversible process, so it should always be used with caution. As per my experience, it is a good approach to hit the GET API after executing the DELETE request to check that the specified resource was deleted from the system successfully. Happy testing!

By Faisal Khatri CORE

As enterprises expand their software development practices and scale their DevOps pipelines, effective management of continuous integration (CI) and continuous deployment (CD) processes becomes increasingly important. GitHub, as one of the most widely used source control platforms, plays a central role in modern development workflows. It is critical for managing code repositories, automating tasks, and enabling collaboration among development teams. To streamline CI/CD activities and ensure smoother operations, many organizations implement a centralized GitHub admin account that oversees repository management, integrations, and automation. This method helps maintain control and consistency across development environments. However, such an approach can introduce security vulnerabilities, scalability challenges, and operational risks, particularly when it comes to handling increasing complexity and ensuring high availability. In this article, we will explore the risks associated with centralized GitHub management, particularly for CI/CD processes, and propose a solution that integrates Jenkins GitHub App authentication and load balancing to address these concerns. By leveraging these strategies, enterprises can enhance security, enable seamless scalability, and reduce the risk of single points of failure. Understanding Centralized GitHub Admin Accounts in Enterprise CI/CD Workflows A centralized GitHub admin account is often used in enterprises to manage critical GitHub repositories, workflows, and integrations across multiple teams. This single account controls access to various repositories and is responsible for executing automation tasks such as continuous integration, deployment, and testing. With centralized administration, it becomes easier to enforce security policies, monitor activity, and apply configuration updates uniformly. However, this approach introduces several challenges: 1. Security Risks Single point of access: Centralized admin accounts provide a single point of entry for all integrations and access to repositories. If compromised, attackers can potentially gain access to sensitive code, secrets, and integrations.Credential management: Centralized management typically requires multiple automation tools and systems to authenticate using shared credentials, increasing the risk of credential theft or misuse. 2. Scalability Limitations Growing demands: As an organization scales, managing access for an increasing number of repositories and integrations using a single account becomes cumbersome. It may lead to slower response times, more complex configuration management, and difficulties in maintaining efficient workflows.Overloaded admin account: When a single admin account is tasked with managing numerous tasks across a large enterprise, it can be overwhelmed, leading to delays in actions and a lack of flexibility. 3. Operational Continuity Concerns Single point of failure: If the centralized admin account encounters issues (e.g., credential errors, revoked access, or network failures), the entire CI/CD pipeline may be disrupted, affecting all dependent workflows.Risk of human error: As the account grows in responsibility, human errors in managing repository configurations or automation scripts can have widespread implications across multiple development teams. Mitigating the Risks With Jenkins GitHub App Authentication To address these risks, enterprises can adopt a more robust and scalable approach to managing GitHub access, using Jenkins GitHub App Authentication as a primary method of securing interactions between GitHub and Jenkins, the popular CI/CD automation tool. How Jenkins GitHub App Authentication Works Jenkins GitHub App Authentication allows Jenkins to securely integrate with GitHub by using an application-specific OAuth token rather than shared credentials. This method provides several benefits: Improved security: Instead of relying on a centralized admin account, Jenkins uses GitHub’s App-based authentication, which gives each Jenkins instance its own token with tightly scoped permissions. This reduces the risk of a single point of compromise and allows fine-grained control over access to GitHub repositories.Granular permissions: With GitHub Apps, access rights can be restricted to specific repositories, branches, or even specific actions (such as push events or pull request interactions). This minimizes unnecessary access and limits the attack surface.Reduced dependency on centralized credentials: GitHub App authentication eliminates the need for central administration accounts to manage credentials across multiple systems, significantly enhancing security posture. Benefits of Jenkins GitHub App Authentication Seamless integration: Jenkins GitHub App authentication streamlines the communication between Jenkins and GitHub, reducing setup complexity and ensuring smoother workflows.Centralized control with distributed management: By leveraging GitHub Apps, organizations can distribute authentication responsibilities across different Jenkins instances, avoiding the bottleneck of a single centralized admin account. Example: Adding GitHub App Credentials to Jenkins Credential Store Below is an example of how to add GitHub App credentials to the Jenkins credential store for secure authentication. Step 1: Create a GitHub App First, create a GitHub App in your GitHub organization. Follow these steps: Navigate to your GitHub organization.Go to Settings > Developer Settings > GitHub Apps and click New GitHub App.Fill in the details for the app and configure the necessary permissions, such as access to repositories, pull requests, and more. Application details: GitHub App name: Choose a name for your GitHub App (e.g., Jenkins GitHub Integration).Homepage URL: Provide a homepage URL (e.g., https://jenkins.example.com).Callback URL: This is the URL where GitHub will send requests to Jenkins. For now, you can leave this blank.Webhook URL: This URL is used to send events from GitHub to Jenkins (typically, http://<jenkins-url>/github-webhook/).Permissions: Under Repository permissions, select the required permissions: Repository contents (Read and Write): For Jenkins to access the contents of your repositories.Pull requests (Read & Write): If you want Jenkins to trigger builds on PR events.Commit statuses (Read & Write): To update commit statuses in GitHub.Under Organization permissions, choose whether the app should have access to the entire organization or just specific repositories.After creating the app, generate a Private Key and download it. You will need this private key for Jenkins authentication. After creating the app, you will be prompted to install it either on individual repositories or on an entire organization. Select the appropriate repositories or organization. Step 2: Install the GitHub Plugin in Jenkins In Jenkins, go to Manage Jenkins > Manage Plugins.Search for and install the GitHub Branch Source plugin. This plugin enables Jenkins to authenticate with GitHub using GitHub Apps. Step 3: Add the GitHub App Credentials to Jenkins In Jenkins, go to Manage Jenkins > Manage Credentials.Choose the appropriate domain (or create a new one) and click Add Credentials.Select GitHub App as the credential type.Fill in the necessary details: GitHub App name: This is the name of the GitHub App you created earlier.GitHub App ID: You can find this ID on the GitHub App settings page.Private Key: Upload the private key you downloaded earlier when creating the GitHub App.Installation ID: This is provided when you install the GitHub App on your repositories or organization.Click Test Connection to verify that Jenkins can connect to GitHub using the credentials you just added. Step 4: Configure Jenkins Job to Use GitHub App Credentials In your Jenkins job configuration, navigate to the Source Code Management section.Select Git and enter your repository URL.Choose the GitHub App credentials you created earlier from the Credentials dropdown. By following these steps, Jenkins will securely authenticate by generating a temporary authentication token for GitHub using the GitHub App credentials, removing the need for a centralized admin account, and the authentication token will be valid only for one hour. Jenkins will also display the remaining rate limit for the GitHub app. Expanding on the Benefits of Temporary Tokens and GitHub App Private Key Security Temporary Tokens for Enhanced Security One of the key benefits of using Jenkins GitHub App authentication is the use of temporary tokens. When Jenkins requests access to GitHub, it does so through a temporary token generated by the GitHub App. This temporary token has the following advantages: Reduced exposure: Since these tokens are short-lived (e.g., they expire after a few hours or minutes), they reduce the window of opportunity for attackers to exploit compromised credentials. This makes it more difficult for malicious actors to use stolen credentials to gain access to GitHub repositories.Automated token refresh: Jenkins automatically manages the renewal of these temporary tokens, ensuring that the authentication process remains seamless without requiring manual intervention.Granular permissions: The temporary token inherits the same granular permissions as the GitHub App, so it has limited access to only the repositories and actions that the app is permitted to perform. GitHub App Private Key Security The private key used by the GitHub App is a critical element of the authentication process. Here’s why managing the GitHub App private key securely is paramount: High sensitivity: The private key is the only means by which Jenkins can authenticate to GitHub via the GitHub App. If compromised, an attacker could potentially impersonate the app and access GitHub repositories.Minimal exposure: The private key should never be exposed to the public or stored in insecure locations. To ensure its security, Jenkins stores the private key in its internal credential store, where access is tightly controlled and monitored.Encryption at rest: Jenkins encrypts credentials stored in its credential store, including the GitHub App private key. This ensures that even if the Jenkins instance is compromised, the sensitive keys remain protected. By securing the private key and ensuring it is never exposed in logs or configuration files, enterprises can significantly mitigate the risk of unauthorized access to their GitHub repositories. Scaling GitHub Access With Load Balancing for GitHub Apps In addition to securing access through Jenkins GitHub App Authentication, enterprises can improve the scalability and resilience of their CI/CD pipelines by employing load balancing for GitHub Apps. This strategy distributes traffic and tasks across multiple GitHub Apps and Jenkins instances to prevent bottlenecks and single points of failure. How Load Balancing Works Load balancing involves distributing the workload of managing GitHub access among several instances of Jenkins and multiple GitHub Apps, rather than depending on a single instance. In a typical load-balanced setup: Multiple Jenkins instances are configured to manage different repositories or workflows.Each Jenkins instance integrates with different GitHub Apps, ensuring that tasks like build triggers, deployment, and testing are not confined to one GitHub App. Benefits of Load Balancing Scalability: As an organization scales, new Jenkins instances and GitHub Apps can be added seamlessly to handle the increased volume of CI/CD activities. This ensures that the pipeline remains fast and responsive even as the number of repositories and integrations grows.Fault tolerance: Load balancing enhances resilience by distributing traffic. If one Jenkins instance or GitHub App experiences issues, the remaining instances can continue managing operations without disruption.Optimized performance: By distributing tasks evenly across multiple instances, the CI/CD process becomes more efficient, with reduced response times and minimized risks of overloading any single component. Strategic Benefits and Implementation 1. Enhanced Security By using Jenkins GitHub App authentication, enterprises can ensure that each Jenkins instance has its own authentication token with fine-grained permissions, drastically reducing the attack surface and improving the overall security model. 2. Improved Scalability Load balancing and distributed GitHub App management make it easier to scale GitHub access as the organization grows, ensuring that new repositories and automation tasks can be efficiently handled by multiple instances without overloading a centralized account. 3. Increased Operational Continuity By spreading the management of GitHub access across multiple instances and apps, enterprises can mitigate the risks of a single point of failure. This ensures that even if one part of the system encounters an issue, the rest of the CI/CD pipeline remains intact and operational. 4. Flexible and Efficient Management With a distributed model, teams can manage their own GitHub Apps and Jenkins instances, giving them more control over their workflows while maintaining overall enterprise-level security and scalability. Conclusion As enterprises scale their development practices and adopt more complex CI/CD processes, ensuring secure, scalable, and resilient management of GitHub access becomes increasingly important. By replacing centralized GitHub admin accounts with Jenkins GitHub App authentication and implementing load balancing across multiple GitHub Apps, enterprises can significantly reduce security risks, improve scalability, and enhance operational continuity. This combination of strategies ensures a more secure and efficient workflow, capable of handling the demands of modern software development at scale.

Operators are a powerful way to extend Kubernetes functionality by acting as custom controllers. They leverage the Kubernetes control loop to manage application lifecycles using declarative custom resources. In this guide, we’ll create a simple “Hello” Operator with the Operator SDK, deploy it on Minikube, and see it in action. Prerequisites Before we begin, make sure you have the following installed and set up on your machine: Minikube You can grab Minikube from the official docs. Start Minikube by running: Shell minikube start Verify the Minikube cluster: PowerShell kubectl cluster-info Go (1.19 or Later) You can download and install Go from the official website. Operator SDK Follow the Operator SDK installation docs to install. Confirm the version: Shell operator-sdk version Docker (Or Similar Container Runtime) We’ll use Docker for building and pushing container images. With these tools in place, you’re all set for a smooth Operator development experience. Project Setup and Initialization Create a Project Directory Let’s begin by creating a fresh directory for our project: Shell mkdir sample-operator-project cd sample-operator-project Initialize the Operator Next, we’ll initialize our Operator project using the Operator SDK. This command scaffolds the basic project layout and configuration files: Shell operator-sdk init --domain=example.com --repo=github.com/youruser/sample-operator Here’s what the flags mean: --domain=example.com sets the domain for your Custom Resources.--repo=github.com/youruser/sample-operator determines the Go module path for your code. You’ll see a freshly generated project structure: Shell sample-operator-project/ ├── Makefile ├── PROJECT ├── go.mod ├── go.sum ├── config/ │ └── ... ├── hack/ │ └── boilerplate.go.txt ├── main.go └── ... Creating the API and Controller Add Your API (CRD) and Controller Our next step is to create the Custom Resource Definition (CRD) and its associated controller. We’ll make a resource called Hello under the group apps and version v1alpha1: Shell operator-sdk create api --group apps --version v1alpha1 --kind Hello --resource --controller This command generates: A new API package under api/v1alpha1/A controller source file in controllers/hello_controller.go Define the Hello CRD Open the file api/v1alpha1/hello_types.go. You’ll see the Hello struct representing our custom resource. We can add a simple Message field to the Spec and a LastReconcileTime field to the Status: Go package v1alpha1 import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) // HelloSpec defines the desired state of Hello type HelloSpec struct { // Message is the text we want our operator to manage. Message string `json:"message,omitempty"` } // HelloStatus defines the observed state of Hello type HelloStatus struct { // Stores a timestamp or an echo of the message LastReconcileTime string `json:"lastReconcileTime,omitempty"` } //+kubebuilder:object:root=true //+kubebuilder:subresource:status // Hello is the Schema for the hellos API type Hello struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"` Spec HelloSpec `json:"spec,omitempty"` Status HelloStatus `json:"status,omitempty"` } //+kubebuilder:object:root=true // HelloList contains a list of Hello type HelloList struct { metav1.TypeMeta `json:",inline"` metav1.ListMeta `json:"metadata,omitempty"` Items []Hello `json:"items"` } Once you’re done, run: Shell make generate make manifests make generate regenerates deepcopy code, and make manifests updates your CRDs in the config/ directory. Implementing the Controller Open controllers/hello_controller.go. The core function here is Reconcile(), which defines how your Operator “reacts” to changes in Hello resources. Below is a minimal example that logs the message and updates LastReconcileTime: Shell package controllers import ( "context" "fmt" "time" "github.com/go-logr/logr" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" appsv1alpha1 "github.com/youruser/sample-operator/api/v1alpha1" ) // HelloReconciler reconciles a Hello object type HelloReconciler struct { client.Client Log logr.Logger } //+kubebuilder:rbac:groups=apps.example.com,resources=hellos,verbs=get;list;watch;create;update;patch;delete //+kubebuilder:rbac:groups=apps.example.com,resources=hellos/status,verbs=get;update;patch //+kubebuilder:rbac:groups=apps.example.com,resources=hellos/finalizers,verbs=update func (r *HelloReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { log := r.Log.WithValues("hello", req.NamespacedName) // Fetch the Hello resource var hello appsv1alpha1.Hello if err := r.Get(ctx, req.NamespacedName, &hello); err != nil { // Resource not found—likely it was deleted log.Info("Resource not found. Ignoring since object must be deleted.") return ctrl.Result{}, client.IgnoreNotFound(err) } // Print the message from Spec log.Info(fmt.Sprintf("Hello Message: %s", hello.Spec.Message)) // Update status with current time hello.Status.LastReconcileTime = time.Now().Format(time.RFC3339) if err := r.Status().Update(ctx, &hello); err != nil { log.Error(err, "Failed to update Hello status") return ctrl.Result{}, err } // Requeue after 30 seconds for demonstration return ctrl.Result{RequeueAfter: 30 * time.Second}, nil } // SetupWithManager sets up the controller with the Manager. func (r *HelloReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). For(&appsv1alpha1.Hello{}). Complete(r) } This snippet ensures each time the custom resource changes, the operator logs a message and updates the status to reflect the time it was last reconciled. Building and Deploying the Operator Set the Container Image In the Makefile, locate the line: CMake IMG ?= controller:latest Replace it with your desired image name (e.g., a Docker Hub repo): CMake IMG ?= your-docker-username/sample-operator:latest Build and Push To build and push your operator image: CMake make docker-build docker-push docker-build compiles your Operator code into a Docker image.docker-push pushes it to your specified image repository. Deploy Onto Minikube Install CRDs: CMake make install This applies your CRD manifests to the cluster. Deploy operator: CMake make deploy This command sets up the operator in a dedicated namespace (usually <project>-system), creates a Deployment, and configures RBAC rules. Check that your deployment is running: Shell kubectl get deployments -n sample-operator-system You should see something like: Shell NAME READY UP-TO-DATE AVAILABLE AGE sample-operator-controller-manager 1/1 1 1 1m Testing Your Operator Create a Hello Resource We’ll now create a sample custom resource to watch the operator in action. Create a file named hello-sample.yaml with the following content: YAML apiVersion: apps.example.com/v1alpha1 kind: Hello metadata: name: hello-sample spec: message: "Hello from my first Operator!" Next, apply the resource: Shell kubectl apply -f hello-sample.yaml Check the CRD's Status Shell kubectl get hellos You should see something like the following: Shell NAME AGE hello-sample 5s Verify Logs and Status Take a look at the operator’s logs: Shell kubectl get pods -n sample-operator-system # Identify the sample-operator-controller-manager pod name, then: kubectl logs sample-operator-controller-manager-xxxxx -n sample-operator-system --all-containers Next, you should see something like: Shell 1.590372e+09 INFO controllers.Hello Hello Message: Hello from my first Operator! You can also inspect the resource’s status: Shell kubectl get hello hello-sample -o yaml Final Project Layout Here’s how your project folder might look at this point: Shell sample-operator-project/ ├── Makefile ├── PROJECT ├── config/ │ ├── crd/ │ │ └── bases/ │ │ └── apps.example.com_hellos.yaml │ ├── default/ │ │ └── kustomization.yaml │ ├── manager/ │ │ ├── kustomization.yaml │ │ └── manager.yaml │ ├── rbac/ │ │ ├── cluster_role.yaml │ │ └── role.yaml │ ├── samples/ │ │ └── apps_v1alpha1_hello.yaml │ └── ... ├── api/ │ └── v1alpha1/ │ ├── hello_types.go │ ├── groupversion_info.go │ └── zz_generated.deepcopy.go ├── controllers/ │ └── hello_controller.go ├── hack/ │ └── boilerplate.go.txt ├── hello-sample.yaml ├── go.mod ├── go.sum └── main.go Conclusion You have just developed a simple Kubernetes Operator that watches a Hello custom resource, prints its message into the logs, and changes its status every time it reconciles. On top of this basic foundation, you can extend the behavior of your Operator for real-world scenarios: managing external services, complex application lifecycles, or advanced configuration management. Operators natively bring Kubernetes management to anything — from applications to infrastructure. With the Operator SDK, everything you need to rapidly scaffold, build, deploy, and test your custom controller logic is right at your fingertips. Experiment with iteration, adapt — and then let automation take over in an operator-driven environment!

By Sai Sandeep Ogety CORE

Air pollution has become a pressing issue in South Asia with recent reports from major news outlets, such as the BBC and CNN, highlighting Lahore in Pakistan as a city struggling with critically poor air quality. Using open data from IQAir, this article plots the Air Quality Index (AQI) for Lahore and expands the analysis to include other locations within Punjab, covering cities in both Pakistan and India. By visualizing AQI levels across these areas, we can shed light on the air pollution crisis, its implications, and potential areas for environmental intervention. The notebook file used in this article is available on GitHub. Introduction Air pollution has emerged as a significant environmental and health concern across South Asia, particularly in densely populated urban areas. Among the most affected cities is Lahore in Pakistan, which has repeatedly been ranked as one of the most polluted cities globally. Recent reports from several news outlets have highlighted Lahore's ongoing struggle with hazardous air quality levels, drawing attention to the urgent need for solutions. In this article, we use data from IQAir to examine the Air Quality Index (AQI) not only in Lahore but also in other cities across Punjab, in both Pakistan and India. By analyzing AQI data across these regions, we can gain insights into the scope of the pollution crisis and identify areas where targeted interventions could help mitigate its impact. Create a SingleStore Cloud Account A previous article showed the steps to create a free SingleStore Cloud account. We'll use the Standard Tier and take the default name for the Workspace Group and Workspace. The Standard Tier is required as, at the time of writing this article, only the Standard Tier currently supports writing files to Stage, which we'll need for saving our Folium map so that we can download it. We'll use the free Community API plan at IQAir for our weather and pollution data. We'll store our IQAir API Key in the secrets vault using the name IQAIR_API_KEY. Import the Notebook We'll download the notebook from GitHub (linked in the introduction). From the left navigation pane in the SingleStore cloud portal, we'll select DEVELOP > Data Studio. In the top right of the web page, we'll select New Notebook > Import From File. We'll use the wizard to locate and import the notebook we downloaded from GitHub. Run the Notebook After checking that we are connected to our SingleStore workspace, we'll run the cells one by one. We'll begin by installing the necessary libraries and importing dependencies. We'll then load our IQAir API Key: Python api_key = get_secret("IQAIR_API_KEY") Next, we'll define the countries and state: Python # Define the countries and state countries = ["Pakistan", "India"] state = "Punjab" # Initialise the list to store locations in the specified tuple format locations = [] Now, we'll loop through each country finding the cities using the IQAir API: Python # Loop through each country for country in countries: # Fetch cities for the specified state and country response_cities = requests.get( f"http://api.airvisual.com/v2/cities?state={state}&country={country}&key={api_key}" ) cities_data = response_cities.json() # Check if the request was successful if cities_data["status"] == "success": # Create each tuple and add it to the locations list locations.extend([(country, state, city["city"]) for city in cities_data["data"]]) else: print(f"Error fetching cities for {state} in {country}: {cities_data.get('data', 'No additional information')}") # Wait before the next API call to avoid rate limits time.sleep(12) We'll now print the list of cities: Python # Print the locations list to verify print(tabulate( locations, headers = ["Country", "State", "City"], tablefmt = "pretty", stralign = "left" )) Example output: Plain Text +----------+--------+-----------------+ | Country | State | City | +----------+--------+-----------------+ | Pakistan | Punjab | Bahawalpur | | Pakistan | Punjab | Lahore | | Pakistan | Punjab | Lodhran | | Pakistan | Punjab | Mangla | | Pakistan | Punjab | Multan | | Pakistan | Punjab | Pindi Bhattian | | Pakistan | Punjab | Rahim Yar Khan | | Pakistan | Punjab | Rawalpindi | | Pakistan | Punjab | Rojhan | | India | Punjab | Amritsar | | India | Punjab | Bathinda | | India | Punjab | Doburji | | India | Punjab | Fatehgarh Sahib | | India | Punjab | Jalandhar | | India | Punjab | Ludhiana | | India | Punjab | Patiala | | India | Punjab | Salodi | +----------+--------+-----------------+ Sometimes weather stations may not be available in some cities, so the number of cities may vary from run to run. Next, we'll prepare two DataFrames to receive the weather and pollution data, as follows: Python weather_columns = [ "Timestamp", "Country", "State", "City", "Latitude", "Longitude", "Temperature", "Pressure", "Humidity", "Wind Speed", "Wind Direction" ] pollution_columns = [ "Timestamp", "Country", "State", "City", "Latitude", "Longitude", "AQI US", "AQI CN" ] weather_df = pd.DataFrame(columns = weather_columns) pollution_df = pd.DataFrame(columns = pollution_columns) Now, we'll iterate over the locations and build up the DataFrames: Python # Iterate over locations to fetch and add data to the dataframes for location in locations: country, state, city = location try: # Fetch data from the API response_api = requests.get( f"https://api.airvisual.com/v2/city?city={city}&state={state}&country={country}&key={api_key}" ) response_api.raise_for_status() parsed_json = response_api.json() dt = datetime.now().strftime("%Y-%m-%d %H:%M:%S") # Extract coordinates coordinates = parsed_json.get("data", {}).get("location", {}) latitude = coordinates.get("coordinates", [np.nan, np.nan])[1] longitude = coordinates.get("coordinates", [np.nan, np.nan])[0] # Extract weather data weather = parsed_json.get("data", {}).get("current", {}).get("weather", {}) temperature = weather.get("tp", np.nan) pressure = weather.get("pr", np.nan) humidity = weather.get("hu", np.nan) wind_speed = weather.get("ws", np.nan) wind_direction = weather.get("wd", np.nan) weather_data = [dt, country, state, city, latitude, longitude, temperature, pressure, humidity, wind_speed, wind_direction] # Append the weather data to the weather dataframe weather_df = pd.concat([weather_df, pd.DataFrame([weather_data], columns = weather_df.columns)], ignore_index = True) # Extract pollution data pollution = parsed_json.get("data", {}).get("current", {}).get("pollution", {}) aqi_us = pollution.get("aqius", np.nan) aqi_cn = pollution.get("aqicn", np.nan) pollution_data = [dt, country, state, city, latitude, longitude, aqi_us, aqi_cn] # Append the pollution data to the pollution dataframe pollution_df = pd.concat([pollution_df, pd.DataFrame([pollution_data], columns = pollution_df.columns)], ignore_index = True) # Add delay to avoid hitting the rate limit # 5 requests every 60 seconds, therefore 1 request every 12 seconds time.sleep(12) except requests.exceptions.RequestException as e: print(f"Failed to fetch data for {city}, {state}: {e}") We'll now create our database: SQL DROP DATABASE IF EXISTS iqair_demo; CREATE DATABASE IF NOT EXISTS iqair_demo; And then create the connection: Python from sqlalchemy import * db_connection = create_engine(connection_url) At this point we'll write the weather and pollution data to SingleStore, which will allow us to perform further analysis, if required: Python weather_df.to_sql( "weather", con = db_connection, if_exists = "replace", index = False, chunksize = 1000 ) pollution_df.to_sql( "pollution", con = db_connection, if_exists = "replace", index = False, chunksize = 1000 ) Now we'll use Plotly Express to create a pollution map: Python # Ensure AQI US is numeric pollution_df["AQI US"] = pd.to_numeric(pollution_df["AQI US"], errors = "coerce") # Now plot the map fig = px.scatter_mapbox( pollution_df, lat = "Latitude", lon = "Longitude", color = "AQI US", size = "AQI US", color_continuous_scale = "OrRd", size_max = 20, zoom = 5, title = "Air Quality Index (AQI) in Punjab, Pakistan-India", hover_name = "City", hover_data = { "AQI US": True, "Timestamp": True } ) # Set map style and layout options fig.update_layout( mapbox_style = "carto-positron", margin = {"r":0, "t":40, "l":0, "b":0} ) fig.show() Example output is shown in Figure 1. Figure 1: AQI in Punjab With some slight modifications, we can use an AQI US color scheme: Python # Define the ranges for each AQI category aqi_bounds = [0, 51, 101, 151, 201, 301, 500] # Define the names of each AQI category aqi_categories = [ "Good", "Moderate", "Unhealthy for Sensitive Groups", "Unhealthy", "Very Unhealthy", "Hazardous" ] # Define the colors for each AQI category aqi_colors = { "Good": "green", "Moderate": "yellow", "Unhealthy for Sensitive Groups": "orange", "Unhealthy": "red", "Very Unhealthy": "purple", "Hazardous": "maroon" } # Map AQI values to categories based on bounds def get_aqi_category(value): for i, bound in enumerate(aqi_bounds): if value < bound: return aqi_categories[i - 1] return aqi_categories[-1] # Apply category mapping to your DataFrame pollution_df["AQI Category"] = pollution_df["AQI US"].apply(get_aqi_category) # Now plot the map using the AQI categories with the discrete color scale fig = px.scatter_mapbox( pollution_df, lat = "Latitude", lon = "Longitude", color = "AQI Category", color_discrete_map = aqi_colors, size = "AQI US", size_max = 20, zoom = 5, title = "Air Quality Index (AQI) in Punjab, Pakistan-India", hover_name = "City", hover_data = { "AQI US": True, "Timestamp": True }, category_orders = { "AQI Category": [ "Hazardous", "Very Unhealthy", "Unhealthy", "Unhealthy for Sensitive Groups", "Moderate", "Good" ] } ) # Set map style and layout options fig.update_layout( mapbox_style = "carto-positron", margin = {"r": 0, "t": 40, "l": 0, "b": 0} ) fig.show() Example output is shown in Figure 2. Figure 2: AQI in Punjab using the US Color Scheme Clearly, for this particular data snapshot, pollution levels are very high. Finally, we'll create a map using Folium: Python # Centered on Lahore, Punjab Lahore = [31.549722, 74.343611] m = folium.Map( location = Lahore, control_scale = True, zoom_start = 6 ) # Prepare the data for the heatmap heat_data = [ [row["Latitude"], row["Longitude"], row["AQI US"]] for index, row in pollution_df.iterrows() ] # Add the heatmap layer HeatMap( data = heat_data, ).add_to(folium.FeatureGroup(name = "AQI Heatmap").add_to(m)) markers_group = folium.FeatureGroup(name = "AQI Markers").add_to(m) # Add custom markers for index, row in pollution_df.iterrows(): folium.Marker( location = [row["Latitude"], row["Longitude"]], popup = f'{row["City"]} AQI US: {row["AQI US"]}', icon = folium.Icon(icon = "info-sign") ).add_to(markers_group) folium.LayerControl().add_to(m) plugins.Fullscreen( position = "topright", title = "Fullscreen", title_cancel = "Exit", force_separate_button = True ).add_to(m) html_content = m._repr_html_() We'll save the map to Stage and then we can download it locally: Python with nb.stage.open("map.html", "w") as st: st.write(html_content) Example output is shown in Figure 3. Figure 3: AQI using Folium Summary In this article, we've plotted the air quality across various cities in Punjab, Pakistan, and India, with a particular focus on Lahore, which has been frequently reported in the media for its poor air quality. By plotting the data from IQAir, the AQI for Lahore and other cities in the region was visualized through several maps. The results highlighted the widespread impact of air pollution across the region, showing that cities in both Pakistan and India often experience unhealthy levels of pollution. Environmental interventions are required and should be prioritized to address the growing concerns over public health and sustainability.

By Akmal Chaudhri CORE

With organizations shifting at a rapid pace to the cloud, securing the infrastructure is of paramount importance in their list of priorities. Even though AWS provides a varied set of tools and services related to security and compliance. There are various other factors beyond security. Security is not just about tools but about strategy, vigilance, continuous improvement, and conformity to the industry compliance standards for secure environments, including GDPR, HIPAA, and PCI DSS. In this article we will discuss AWS security components with best practices based on a deep-down analysis. AWS Security Components AWS has a rich set of security tools for strengthening cloud environments. At the core of AWS security is a shared responsibility model, which clearly defines responsibilities between customers and AWS. AWS provides cloud infrastructure security while customers handle data and configurations. This demarcation constitutes the core of AWS security practices with some of the key security components including: AWS Identity and Access Management (IAM) IAM manages access to the AWS resources with fine-grained permissions. Least privileges are recommended to decrease security risks. AWS Security Hub AWS Security Hub provides an aggregated view of compliance and security posture, creating findings from services such as AWS Config, GuardDuty, and Inspector. AWS Key Management Service (KMS) AWS KMS manages the encryption keys, assuring in-transit safe data storage. Amazon GuardDuty AWS GuardDuty provides a threat detection service leveraging machine learning to scan logs for potential threats. AWS Config This service continuously monitors and evaluates configurations of AWS resources against specified compliance standards. AWS Security Workflow A typical flow for AWS security components begins with logging and auditing through CloudTrail and CloudWatch Logs. Events that trigger alerts are sent to AWS Security Hub, where actionable insights are derived. Threats identified by GuardDuty could trigger automation workflows through AWS Lambda that could result in isolating compromised resources or triggering the response team notifications. While these components work in tandem, an organization's strategy and practices deployed will have a great impact on deployment. AWS Security Analysis and Best Practices While carrying out our analysis, including AWS whitepapers, customer case studies, and security incidents, some trends appear that are common pitfalls and best practices that can be put into action. Vulnerabilities in "Lift and Shift" Strategies Most organizations assume that their on-premise security strategies only apply to the cloud. Statistics point out that this assumption leads to misconfigurations, which is the leading cause of security incidents in AWS. For example, improper S3 bucket configuration is given as the reason for some high-profile data breaches. (Source: Gartner). Best Practices Manage isolation between AWS and other cloud environments (if applicable). AWS Config can be leveraged to enforce compliance checks on S3 bucket policies and other resources. Prioritize Identity and Access Management According to a Verizon Data Breach Investigations Report, more than 70% of breaches stem from mismanaged credentials. Furthermore, many organizations seemingly grant IAM roles with access that is too broad simply because it's hard to configure strict IAM roles. Best Practices Use the principle of least privilege for IAM roles and users. Ensure IAM Access Analyzer has identified excessive permissions.For privileged accounts, enforce MFA. Leverage Infrastructure as Code Manual configurations can be a source of drift and offer many opportunities for human error to occur. AWS CloudFormation can be used to define sets of secure templates for infrastructure deployment. Best Practices Security baselines can be defined within IaC templates and then injected into the CI/CD pipeline. Use AWS CodePipeline to enforce code reviews and security checks on deployment. Implement Threat Detection Mechanisms Many organizations underutilize threat detection mechanisms, whether due to the difficulty or expense. In some cases, enabling Amazon GuardDuty and AWS Macie has been shown to greatly improve response times (Source: AWS Security Blog). Best Practices Enable GuardDuty and tune it to alert the security team in a timely manner.Regularly run threat simulation exercises to test their response. Data Encryption and Monitoring AWS Docs highlighted that data encryption is seen as an approach that is "set it and forget it," which causes old or badly managed encryption keys. Organizations using continuous monitoring with CloudTrail with the help of regular penetration have a higher chance of pre-vulnerability detection. The approach aligns with the 2024 Verizon Data Breach Investigations Report (DBIR), findings that highlight monitoring importance and management. Best Practices Using AWS KMS for all encryption with automatic key rotation policies Continuously monitor account activity using Conclusion AWS CloudTrail. The security of the AWS environment is not about putting every component in place; rather, it's about being strategic about reaching your organizational goals and compliance needs. AWS offers many services for a successful, well-informed implementation along with active management. However, our analysis highlights that organizations perceiving cloud security as a journey rather than an event perform better against emerging threats. Organizations using AWS components productively, practicing best practices, and constantly striving for improvement can successfully strengthen the security and compliance of their AWS environments.

In today’s rapidly evolving IT landscape, the ability to respond quickly to system changes is essential. Event-driven automation enables systems to react instantly to specific triggers or events, enhancing infrastructure resilience and efficiency. A simple and effective method for implementing event-driven automation is through webhooks, which can initiate specific actions in response to events. In this article, I’ll walk you through an example of using Ansible to monitor and manage a Nginx web server — specifically, to demonstrate how to use a URL check module to trigger a node restart playbook that will automatically start the Nginx server when a particular URL becomes unreachable. About the Module The ansible.eda.url_check module in Ansible is part of the event-driven automation collection designed to monitor the availability of a specified URL. This module is used to automate the process of checking whether a web application or service is reachable by performing an HTTP request to a given URL. When integrated into event-driven workflows, this module can trigger actions, such as restarting a service or alerting the team, whenever the URL becomes unreachable or encounters issues. In this example, we are verifying the accessibility of the localhost URL, and if it is unreachable, the Nginx restart command is triggered. This demonstration ensures that the Nginx service is automatically restarted whenever the URL becomes unavailable. Step 1 To install Nginx using the brew command, you can run brew install nginx on macOS, which will automatically download and install Nginx along with its dependencies. By default, Homebrew installs Nginx in the directory /usr/local/Cellar/nginx/, and it configures the software for use with macOS systems. After installation, configure Nginx to listen on port 8080 by editing the configuration file located at /usr/local/etc/nginx/nginx.conf, changing the listen directive to listen 8080;, and then starting the Nginx service using brew services start nginx. To verify that Nginx is up and running, open a terminal and execute the command curl http://localhost:8080/. If the server is properly configured, you should receive an HTTP response from Nginx, indicating that it's successfully serving content on port 8080. Step 2 The monitor-nginx.yml file defines the sources and rules for monitoring the Nginx server. The url_check task polls the URL http://localhost:8080 every 30 seconds to verify its status. If the URL is detected as down, the script triggers an action to address the issue. This action involves executing a secondary playbook designed to restart the Nginx server. The setup ensures continuous monitoring and automated recovery of the service. YAML --- - name: Monitor the localhost url, http://localhost:8080 hosts: localhost sources: - ansible.eda.url_check: urls: - http://localhost:8080 delay: 30 rules: - name: Start Nginx server if the status is down condition: event.url_check.status == "down" action: run_playbook: name: restart-server.yml The restart-server.yml script is to handle the task of starting the Nginx server. YAML --- - hosts: localhost gather_facts: false connection: local tasks: - name: localhost url is down, restarting Nginx ansible.builtin.command: brew services start nginx ~ Demo To monitor the Nginx server, execute the ansible-rulebook -i localhost -r monitor-nginx.yml command, where -i localhost refers to the inventory file specifying the target machine (in this case, the local machine). The -r monitor-nginx.yml flag tells Ansible to run the monitor-nginx.yml rulebook, which includes instructions for checking the accessibility of the Nginx server at http://localhost:8080. The rulebook will monitor the server every 30 seconds and trigger the Nginx restart task if the server becomes unreachable. To check the current status of the Nginx service, execute the brew services info nginx command. Once verified, stop the Nginx service managed by Homebrew using the command brew services stop nginx. This action ensures that the Nginx service is properly stopped. Next, repeat the status check to confirm if the monitoring script detects the service-down event. Verify that the script takes appropriate action to restart the service automatically. Conclusion Event-driven automation empowers systems to respond instantly to specific triggers, enhancing the responsiveness and efficiency of IT operations. This demonstration showcased the use of Ansible in conjunction with url_check to automate tasks such as restarting the Nginx server in response to the service-down scenario. By incorporating tools like Ansible.EDA, organizations can build robust event-driven workflows that enhance the agility, resilience, and manageability of your infrastructure. Whether it's automating cloud deployments, managing configuration updates, or scaling services, event-driven automation provides a versatile and modern approach to infrastructure management. This approach ensures that your systems remain adaptable and ready to meet evolving demands with minimal manual intervention. Note: The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.