DevOps and CI/CD

In the world of software development, Agile and DevOps have gained popularity for their focus on efficiency, collaboration, and delivering high-quality products. Although they have different goals, Agile and DevOps are often used interchangeably. This article seeks to illuminate the distinctions and commonalities, between these approaches, demonstrating how they synergize seamlessly to produce results. Figure courtesy of Browser Stack Understanding Agile Overview Agile is a project management and software development methodology that emphasizes an approach, to delivering projects. Emerging from the Agile Manifesto in the 2000s. Agile focuses on working with customers adjusting plans as needed, striving for ongoing enhancements, and making small changes gradually instead of large-scale launches. Key Principles Agile is founded on four principles: Teams value. Communication, above adherence to procedures or tools. Priority is given to creating software, over documentation. Customer engagement and feedback are promoted during the development phase. Adapting to evolving needs is favored over sticking to a predetermined plan. Top Agile Frameworks There are frameworks that have been created based on the following principles: In Scrum, tasks are often broken down into sprints lasting around 2 to 4 weeks with check-ins and evaluations. Kanban on the other hand employs a Kanban board to manage progress and review tasks. Extreme Programming (XP): This technique uses practices such as test driven development, continuous integration, and pair programming to improve software quality. Understanding DevOps Overview DevOps, short, for Development and Operations encompasses practices, cultural values, and tools that promote teamwork, between software development (Dev) and IT operations (Ops). The primary goal of DevOps is to shorten the development cycle, boost deployment frequency, and guarantee the delivery of top-notch software. Key Principles DevOps is driven by the following principles: Fostering teamwork and joint effort is the key to promoting a sense of shared duty, between development and operations teams. By embracing Continuous Integration and Continuous Delivery you can make sure that any changes, to the code are thoroughly tested, integrated, and deployed into environments seamlessly. Emphasizing real-time monitoring, logging, and feedback mechanisms for timely issue identification and resolution. Key Practices in DevOps DevOps revolves around the following core practices: Managing infrastructure configurations through code to automate the setup and control of infrastructure resources is known as Infrastructure, as Code. Continuous Integration involves integrating code changes into a repository, with automated builds and tests to detect any problems quickly. Continuous Delivery builds on CI by automating the deployment process for releasing code changes to production. Automated Testing involves incorporating automated tests at each development phase to uphold code quality and functionality. Comparison Between DevOps and Agile To distinguish between Agile and DevOps it is beneficial to compare them across aspects. Here is a comparison chart summarizing the elements of Agile and DevOps: Objective Agile DevOps Focus Software development and project management Software development and IT operations Primary Goal Delivering small, incremental changes frequently Shortening development lifecycle, improving deployment frequency Core Principles Customer collaboration, adaptive planning, continuous improvement Collaboration, automation, CI/CD, monitoring Team Structure Cross-functional development teams Integrated Dev and Ops teams Frameworks Scrum, Kanban, XP CI/CD, IaC (infrastructure as code), Automated Testing Feedback Loop Iterative feedback from customers Continuous feedback from monitoring and logging Automation Limited focus on automation Extensive automation for builds, tests, and deployments Documentation Lightweight, as needed Comprehensive, includes infrastructure as code Cultural Philosophy Agile mindset and values DevOps culture of collaboration and shared responsibility Implementation Scope Primarily within development teams Across development and operations teams Difference Between Agile and DevOps Agile and DevOps both have objectives in enhancing software delivery and quality. They diverge in various aspects: Scope and Emphasis Agile: Centers, on refining the software development process and project management. It stresses development, customer engagement, and flexibility to accommodate changes. DevOps: Goes beyond development to encompass IT operations striving to enhance the software delivery cycle. DevOps methodologies prioritize collaboration, between development and operations automation and continuous integration and delivery. Team Setup Agile methodology involves teams comprising developers, testers, and business analysts working closely together. While each team member may have roles they collaborate harmoniously to achieve shared objectives. In contrast, DevOps advocates for integrated teams where both development and operations professionals collaborate seamlessly throughout the software delivery lifecycle. This collaborative approach helps break down barriers between teams. Encourages a culture of responsibility. Automation Practices Under practice, tools are used to support development activities; however, the emphasis on automation is not as pronounced as in DevOps. Agile teams may automate tasks like testing but primarily focus on iterative development and customer feedback. DevOps places emphasis on automation as a tenet. By automating build processes testing procedures and deployment tasks DevOps aims to enhance efficiency minimize errors and facilitate delivery. Feedback Channels Agile relies, on receiving feedback from customers and stakeholders through sprint reviews and retrospectives to drive enhancements. DevOps underscores the importance of feedback obtained from monitoring systems and logging mechanisms. DevOps teams leverage real-time data to swiftly identify and address issues ensuring optimal software performance, in production settings. Cultural Philosophy Agile philosophy: Centers on the core values and mindset of Agile, which prioritize collaboration with customers, adaptability, and continuous enhancement. It fosters a culture of flexibility and responsiveness to changes. DevOps culture: Focuses on nurturing an environment of shared responsibility and ongoing learning between development and operations teams. The goal of DevOps is to establish a setting where all team members collaborate towards objectives. Similarities Between Agile and DevOps Despite their variances, Agile and DevOps exhibit resemblances that complement each other: Emphasis on collaboration: Both Agile and DevOps stress the significance of collaboration among team members. Agile encourages functional teamwork while DevOps supports merging development with operations to enhance communication and break down barriers. Continuous enhancement: Both methodologies prioritize processes for improvement. Agile concentrates on delivering changes based on customer feedback while DevOps highlights integration/delivery for rapid enhancements driven by real-time monitoring feedback. Customer-focused approach: Both Agile and DevOps place emphasis, on delivering value to customers. Agile methodologies prioritize working closely with customers. Gathering feedback to ensure that the final product meets user requirements. On the one hand, DevOps practices focus on delivering top-notch software and consistently enhancing the overall customer experience. Embracing change and adaptability: Both Agile and DevOps emphasize the importance of being adaptable, in the development process. Agile encourages teams to be responsive to evolving needs. Adjust their strategies accordingly. Similarly, DevOps empowers teams to swiftly address issues. Make necessary tweaks to enhance performance and reliability. The Verdict? In software development, both Agile and DevOps play huge roles in offering distinct advantages and catering to different aspects of the software delivery lifecycle. While Agile concentrates on refining development processes and project management through practices centered around customer needs DevOps extends these principles by incorporating IT operations stressing more on collaboration, automation, and continuous deployment. When To Use Agile Agile is ideal for projects where: Requirements are expected to change frequently Customer feedback is crucial to the development process The project involves a high degree of complexity and uncertainty Teams need a flexible, iterative approach to manage work When To Use DevOps DevOps is suitable for organizations that: Require frequent, reliable software releases Have a need to improve collaboration between development and operations teams Aim to reduce time to market and enhance deployment frequency Want to implement extensive automation in their build, test, and deployment processes Combining Agile and DevOps Seek collaboration, between development and operations teams. To speed up the time it takes to bring products to market and increase how often they're deployed organizations are aiming for automation, in their building, testing, and deployment procedures. By merging Agile and DevOps methods companies can gain advantages. Agile principles are used for project management and development practices while DevOps practices handle deployment and operations. This combination allows teams to have an effective and top-notch software delivery process. It lets organizations adapt swiftly to changing needs provide value to customers and uphold performance levels in production environments. Conclusion Agile and DevOps are both methodologies that have transformed the software development field. Understanding their distinctions, similarities, and how they work together is vital, for organizations seeking to optimize their software delivery procedures. Capitalizing on the strengths of both Agile and DevOps teams can foster a culture of teamwork ongoing enhancement and customer focus—ultimately delivering top-quality software that meets user expectations. Let me know in the comments which one you use in your company.

By Naga Santhosh Reddy Vootukuri CORE

Striving for "100% Infrastructure as Code" has become a common goal for many DevOps teams today. Defining and managing nearly all infrastructure provisioning and configuration aspects through code is robust. It saves time, reduces manual toil, improves consistency, and minimizes human errors. However, Infrastructure as Code (IaC) is not a panacea for every DevOps process or workflow. As Derek Ashmore, Application Transformation Principal at Asperitas, argues, aiming for 99% IaC coverage is often better than striving for 100%. Knowing when not to use IaC is just as important as leveraging it extensively. The following explores the limitations of IaC and discusses how to identify processes that may be better handled manually, based on the Q&A session I had with Derek. The Limitations of Infrastructure as Code While IaC tooling has matured to the point where it's technically possible to automate virtually any infrastructure process, you can still do so. There are scenarios where an IaC approach can create more problems than it solves: Infrequently performed processes (once or twice a year at most) Workflows that rely on third-party resources outside your control Operations involving resources that can't be easily recreated or relaunched In these situations, the upfront and ongoing costs of developing and maintaining IaC can outweigh the time savings and other benefits it provides. Let's examine each of these in more detail. Evaluating Infrequent Processes For processes only performed once or twice per year, the value of automating them with IaC diminishes. Ashmore uses a simple formula to evaluate the cost-benefit: Assume a 2-year "shelf life" for IaC before it needs significant maintenance. Estimate 30% of initial development time/cost for that maintenance work. Calculate the break-even point when the labor cost of IaC is less than that of manual execution. "For infrequently executed manual processes, the break-even point is never reached," he explains. The labor required to develop, test, and maintain the IaC is at most the toil of executing the process manually, given how rarely it occurs. An SSL certificate renewal that happens every year or two is an example. It's not hard to write code to automate the renewal, but it's also not time-consuming to do manually. In the interim, the certificate authority may change its process, breaking your IaC when you rerun it. Working With Third-Party Dependencies IaC is also problematic for workflows involving resources managed by third parties, like an interconnection between a public cloud and a colocation facility. "You can't fully manage operations you don't control," says Ashmore. In this example, you rely on the colocation provider to establish the physical connection on their schedule. With control over the resource, automating its provisioning end-to-end with IaC is easier. However, Ashmore still advises automating the portions of these workflows you control. "The same principles for evaluating the cost-effectiveness of that IaC apply," he notes. If the time savings of partial automation justify the development and maintenance costs, it may be worth pursuing. Avoiding Resources That Can’t Be Easily Recreated Finally, be cautious about using IaC for resources that can only be recreated as part of an automated workflow. Secrets like passwords and encryption keys are a prime example. While secret management tools often have features for recovering deleted secrets within a time window, they can't typically be undeleted if too much time has passed. This makes re-running your IaC workflows to manage those secrets challenging, which prevents effective iteration and testing. Strategies for Maximizing IaC Value So, how can DevOps teams get the most value from IaC while avoiding these pitfalls? Ashmore offers a few key recommendations: Focus on minimizing manual labor over time, not reaching 100% IaC. "The difference is subtle but significant." Implement automated testing to identify when changes to technologies, APIs, or policies have "broken" your IaC. This allows you to fix issues proactively before they impact the business. Design your IaC templates to be modular and easily maintained. While this will not prevent all maintenance, it will make it less costly. Continually reevaluate your IaC decisions as tools and platforms evolve. A process that doesn't make sense for IaC today may make sense in the future. Conclusion Infrastructure as Code is a game-changer for DevOps when applied smartly. By being selective about where and how you leverage IaC, you can get maximum value from it while avoiding unnecessarily high development and maintenance costs. Remember, the goal is not to reach 100% IaC coverage. It's to minimize manual effort and maximize efficiency overall. Knowing when not to use IaC is a crucial part of achieving that objective.

By Tom Smith CORE

Debugging application issues in a Kubernetes cluster can often feel like navigating a labyrinth. Containers are ephemeral by design and intended to be immutable once deployed. This presents a unique challenge when something goes wrong and we need to dig into the issue. Before diving into the debugging tools and techniques, it's essential to grasp the core problem: why modifying container instances directly is a bad idea. This blog post will walk you through the intricacies of Kubernetes debugging, offering insights and practical tips to effectively troubleshoot your Kubernetes environment. The Problem With Kubernetes Video The Immutable Nature of Containers One of the fundamental principles of Kubernetes is the immutability of container instances. This means that once a container is running, it shouldn't be altered. Modifying containers on the fly can lead to inconsistencies and unpredictable behavior, especially as Kubernetes orchestrates the lifecycle of these containers, replacing them as needed. Imagine trying to diagnose an issue only to realize that the container you’re investigating has been modified, making it difficult to reproduce the problem consistently. The idea behind this immutability is to ensure that every instance of a container is identical to any other instance. This consistency is crucial for achieving reliable, scalable applications. If you start modifying containers, you undermine this consistency, leading to a situation where one container behaves differently from another, even though they are supposed to be identical. The Limitations of kubectl exec We often start our journey in Kubernetes with commands such as: $ kubectl -- exec -ti <pod-name> This logs into a container and feels like accessing a traditional server with SSH. However, this approach has significant limitations. Containers often lack basic diagnostic tools—no vim, no traceroute, sometimes not even a shell. This can be a rude awakening for those accustomed to a full-featured Linux environment. Additionally, if a container crashes, kubectl exec becomes useless as there's no running instance to connect to. This tool is insufficient for thorough debugging, especially in production environments. Consider the frustration of logging into a container only to find out that you can't even open a simple text editor to check configuration files. This lack of basic tools means that you are often left with very few options for diagnosing problems. Moreover, the minimalistic nature of many container images, designed to reduce their attack surface and footprint, exacerbates this issue. Avoiding Direct Modifications While it might be tempting to install missing tools on the fly using commands like apt-get install vim, this practice violates the principle of container immutability. In production, installing packages dynamically can introduce new dependencies, potentially causing application failures. The risks are high, and it's crucial to maintain the integrity of your deployment manifests, ensuring that all configurations are predefined and reproducible. Imagine a scenario where a quick fix in production involves installing a missing package. This might solve the immediate problem but could lead to unforeseen consequences. Dependencies introduced by the new package might conflict with existing ones, leading to application instability. Moreover, this approach makes it challenging to reproduce the exact environment, which is vital for debugging and scaling your application. Enter Ephemeral Containers The solution to the aforementioned problems lies in ephemeral containers. Kubernetes allows the creation of these temporary containers within the same pod as the application container you need to debug. These ephemeral containers are isolated from the main application, ensuring that any modifications or tools installed do not impact the running application. Ephemeral containers provide a way to bypass the limitations of kubectl exec without violating the principles of immutability and consistency. By launching a separate container within the same pod, you can inspect and diagnose the application container without altering its state. This approach preserves the integrity of the production environment while giving you the tools you need to debug effectively. Using kubectl debug The kubectl debug command is a powerful tool that simplifies the creation of ephemeral containers. Unlike kubectl exec, which logs into the existing container, kubectl debug creates a new container within the same namespace. This container can run a different OS, mount the application container’s filesystem, and provide all necessary debugging tools without altering the application’s state. This method ensures you can inspect and diagnose issues even if the original container is not operational. For example, let’s consider a scenario where we’re debugging a container using an ephemeral Ubuntu container: kubectl debug <myapp> -it <pod-name> --image=ubuntu --share-process --copy-to=<myapp-debug> This command launches a new Ubuntu-based container within the same pod, providing a full-fledged environment to diagnose the application container. Even if the original container lacks a shell or crashes, the ephemeral container remains operational, allowing you to perform necessary checks and install tools as needed. It relies on the fact that we can have multiple containers in the same pod, that way we can inspect the filesystem of the debugged container without physically entering that container. Practical Application of Ephemeral Containers To illustrate, let’s delve deeper into how ephemeral containers can be used in real-world scenarios. Suppose you have a container that consistently crashes due to a mysterious issue. By deploying an ephemeral container with a comprehensive set of debugging tools, you can monitor the logs, inspect the filesystem, and trace processes without worrying about the constraints of the original container environment. For instance, you might encounter a situation where an application container crashes due to an unhandled exception. By using kubectl debug, you can create an ephemeral container that shares the same network namespace as the original container. This allows you to capture network traffic and analyze it to understand if there are any issues related to connectivity or data corruption. Security Considerations While ephemeral containers reduce the risk of impacting the production environment, they still pose security risks. It’s critical to restrict access to debugging tools and ensure that only authorized personnel can deploy ephemeral containers. Treat access to these systems with the same caution as handing over the keys to your infrastructure. Ephemeral containers, by their nature, can access sensitive information within the pod. Therefore, it is essential to enforce strict access controls and audit logs to track who is deploying these containers and what actions are being taken. This ensures that the debugging process does not introduce new vulnerabilities or expose sensitive data. Interlude: The Role of Observability While tools like kubectl exec and kubectl debug are invaluable for troubleshooting, they are not replacements for comprehensive observability solutions. Observability allows you to monitor, trace, and log the behavior of your applications in real time, providing deeper insights into issues without the need for intrusive debugging sessions. These tools aren't meant for everyday debugging: that role should be occupied by various observability tools. I will discuss observability in more detail in an upcoming post. Command Line Debugging While tools like kubectl exec and kubectl debug are invaluable, there are times when you need to dive deep into the application code itself. This is where we can use command line debuggers. Command line debuggers allow you to inspect the state of your application at a very granular level, stepping through code, setting breakpoints, and examining variable states. Personally, I don't use them much. For instance, Java developers can use jdb, the Java Debugger, which is analogous to gdb for C/C++ programs. Here’s a basic rundown of how you might use jdb in a Kubernetes environment: 1. Set Up Debugging First, you need to start your Java application with debugging enabled. This typically involves adding a debug flag to your Java command. However, as discussed in my post here, there's an even more powerful way that doesn't require a restart: java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar myapp.jar 2. Port Forwarding Since the debugger needs to connect to the application, you’ll set up port forwarding to expose the debug port of your pod to your local machine. This is important as JDWP is dangerous: kubectl port-forward <pod-name> 5005:5005 3. Connecting the Debugger With port forwarding in place, you can now connect jdb to the remote application: jdb -attach localhost:5005 From here, you can use jdb commands to set breakpoints, step through code, and inspect variables. This process allows you to debug issues within the code itself, which can be invaluable for diagnosing complex problems that aren’t immediately apparent through logs or superficial inspection. Connecting a Standard IDE for Remote Debugging I prefer IDE debugging by far. I never used JDB for anything other than a demo. Modern IDEs support remote debugging, and by leveraging Kubernetes port forwarding, you can connect your IDE directly to a running application inside a pod. To set up remote debugging we start with the same steps as the command line debugging. Configuring the application and setting up the port forwarding. 1. Configure the IDE In your IDE (e.g., IntelliJ IDEA, Eclipse), set up a remote debugging configuration. Specify the host as localhost and the port as 5005. 2. Start Debugging Launch the remote debugging session in your IDE. You can now set breakpoints, step through code, and inspect variables directly within the IDE, just as if you were debugging a local application. Conclusion Debugging Kubernetes environments requires a blend of traditional techniques and modern tools designed for container orchestration. Understanding the limitations of kubectl exec and the benefits of ephemeral containers can significantly enhance your troubleshooting process. However, the ultimate goal should be to build robust observability into your applications, reducing the need for ad-hoc debugging and enabling proactive issue detection and resolution. By following these guidelines and leveraging the right tools, you can navigate the complexities of Kubernetes debugging with confidence and precision. In the next installment of this series, we’ll delve into common configuration issues in Kubernetes and how to address them effectively.

By Shai Almog CORE

In the fast-paced world of software development, efficiency and speed play an important role. Setting up a development environment can be a time-consuming task for developers. GitHub Codespaces, a cloud-based environment, aims to address this challenge by offering access to a configured setup. This guide will help you kickstart your journey with GitHub Codespaces and showcase how it can significantly accelerate the environment setup process for developers. What Is GitHub Codespaces? GitHub Codespaces is an all-in-one workspace for developers that provides an integrated development environment (IDE) where they can effortlessly build and access their coding setups from their GitHub repositories. This innovative platform leverages Visual Studio Code (VS Code) in the cloud for a development experience whether you're tackling a project or diving into a complex enterprise application. Advantages of GitHub Codespaces Instant setup: Developers can dive into coding within minutes without having to manually set up all the dependencies. Consistency: All developers work in the environment reducing conflicts arising from the "IT WORKS ON MY MACHINE" issue. Flexibility: Access your workspace by using any device connected to the internet. Scalability: Easily cater to varying project requirements by adjusting resources such, as CPU and memory. Integrated with GitHub: Streamlined connection, with GitHub repositories makes work and collaboration easier. Challenges in Using GitHub Codespaces Initial setup time: For larger repositories, the initial setup time can be significant. Cost: While GitHub Codespaces offers a free plan, larger teams or projects may require paid plans, which significantly increase the overall cost of development. Internet connection: As a cloud-based service, it requires a stable internet connection, which can be a challenge for developers working in areas with poor internet connectivity or during travel. Limited customizations: Though GitHub Codespaces provides a lot of flexibility, it might not support all the customizations that a developer might have on their local machine. Performance: While GitHub Codespaces is designed to be fast and responsive, the performance might not match that of a powerful local machine, especially for resource-intensive tasks. Learning curve: Developers who are accustomed to local developer environments might experience a learning curve when getting used to a cloud-based IDE. Starting Out With GitHub Codespaces Prerequisites Before you dive into using GitHub Codespaces make sure you have the following: A GitHub account (Pro or an organization's paid plan) Permission to access the repository you wish to collaborate on Step-By-Step Instructions Step 1: Activate GitHub Codespaces Go to your repository: Head to the repository where you intend to set up a Codespace. Enable Codespaces: If Codespaces isn't enabled for your account or organization yet, visit the repository settings. Turn it on. Step 2: Set Up a Codespace Create a new Codespace: Click on the "Code" button on the repository page. Select the "Codespaces" tab, where you will see a green button in the center to create Codespaces on main. Configure your workspace: Select the branch and configuration file (devcontainer.json) if its provided. Press "Create Codespace" to begin. Step 3: Personalize Your Development Environment Access VS Code: Once your workspace is prepared, it will launch in a web-based version of Visual Studio Code. Add extensions: Install VS Code extensions from the Extensions Marketplace to enrich your development setup. Adjust your settings: Make any changes to the settings and configurations to align with your development process. Step 4: Commence Coding Figure 1: I have taken Contoso-chat Azure samples GitHub repo to demo Once your Codespace is set up, you can dive into coding. The devcontainer.json file ensures that all required dependencies and tools are already installed, creating a customized environment tailored to suit your project's requirements. Enhancing Development With GitHub Codespaces 1. Pre-Set Development Environments GitHub Codespaces utilizes development containers outlined in a devcontainer.json file. This file defines the setup of the development environment, encompassing the operating system, tools, libraries, and dependencies needed. Below is an example of what a devcontainer.json file may look like: JSON // For format details, see https://aka.ms/devcontainer.json. For config options, see the // README at: https://github.com/devcontainers/templates/tree/main/src/python { "name": "Contoso Chat (v2)", "build": { "dockerfile": "Dockerfile", "context": ".." }, "features": { "ghcr.io/devcontainers/features/azure-cli:1": { "installBicep": true, "extensions": "ml" }, "ghcr.io/devcontainers/features/git:1": {}, "ghcr.io/azure/azure-dev/azd:latest": {}, "ghcr.io/devcontainers/features/docker-in-docker:2": {} }, "customizations": { "vscode": { "extensions": [ "prompt-flow.prompt-flow", "ms-azuretools.vscode-docker", "ms-python.python", "ms-toolsai.jupyter", "ms-azuretools.vscode-bicep", "rogalmic.bash-debug" ] } } } This setup ensures that every Codespace created from this repository will come equipped with Python, Bicep, Docker, etc. set up for use. 2. Smooth Collaboration GitHub Codespaces streamlines collaboration by ensuring that all team members operate within the development environment. Any modifications made to the devcontainer.json file can be saved to the repository promptly updating the environment for everyone. This uniformity reduces setup time and eliminates discrepancies in environments that could lead to bugs and integration challenges. 3. Adaptable Resource Allocation Depending on your project's needs, you can select machine types with varying CPU and memory configurations for your Codespaces. This adaptability ensures that you have the required resources to handle demanding tasks without sacrificing performance. 4. Convenience and Flexibility A standout feature of GitHub Codespaces is its capability to access your development environment from any device. Once you set up a new Codespace, it will show up inside the Codespaces and make it easy to open from any machine. Whether you're working on a desktop, laptop, or tablet you can seamlessly continue your development tasks as long as you are connected to the Internet. This flexibility boosts productivity. Effective GitHub Codespaces Usage Recommendations 1. Utilize Devcontainer.json Efficiently Define dependencies clearly: Ensure all essential dependencies and tools are clearly outlined in the devcontainer.json file. Custom commands: Utilize the feature to execute scripts or commands once the container is created like installing software or configuring databases. Extensions: Pre-install VS Code extensions to improve your coding experience. 2. Efficient Resource Management Select appropriate machine type: Choose a machine type that suits your project requirements. Smaller projects may function well with resources while larger projects might need robust machines. Monitoring resource usage: Keep track of resource consumption. Adjust settings as necessary, for performance. 3. Effective Collaboration Uniform environment setup: Ensure that the devcontainer.json file remains consistent and updated across all team members. Shared configurations: Share configurations and extensions via the repository to maintain a development environment. Conclusion GitHub Codespaces is a tool that simplifies development by offering consistent and scalable environments. By minimizing setup time and configuration hassle, developers can dedicate time to coding rather than managing their environment and prerequisites. Whether working on projects or collaborating with teams GitHub Codespaces, can significantly boost productivity. Getting started with GitHub Codespaces is simple. Its impact on the development process is substantial. Please give it a try if you haven't and share your experience. Happy coding!!

By Naga Santhosh Reddy Vootukuri CORE

So, I’ve always thought about Heroku as just a place to run my code. They have a CLI. I can connect it to my GitHub repo, push my code to a Heroku remote, and bam…it’s deployed. No fuss. No mess. But I had always run my test suite…somewhere else: locally, or with CircleCI, or in GitHub Actions. How did I not know that Heroku has CI capabilities? Do you mean I can run my tests there? Where have I been for the last few years? So that’s why I didn’t know about Heroku CI… CI is pretty awesome. You can build, test, and integrate new code changes. You get fast feedback on those code changes so that you can identify and fix issues early. Ultimately, you deliver higher-quality software. By doing it in Heroku, I get my test suite running in an environment much closer to my staging and production deployments. If I piece together a pipeline, I can automate the progression from passing tests to a staging deployment and then promote that staged build to production. So, how do we get our application test suite up and running in Heroku CI? It will take you 5 steps: Write your tests Deploy your Heroku app Push your code to Heroku Create a Heroku Pipeline to use Heroku CI Run your tests with Heroku CI We’ll walk through these steps by testing a simple Python application. If you want to follow along, you can clone my GitHub repo. Our Python App: Is It Prime? We’ve built an API in Python that listens for GET requests on a single endpoint:/prime/{number}. It expects a number as a path parameter and then returns true or false based on whether that number is a prime number. Pretty simple. We have a modularized function in is_prime.py: Python def is_prime(num): if num <= 1: return False if num <= 3: return True if num % 2 == 0 or num % 3 == 0: return False i = 5 while i * i <= num: if num % i == 0 or num % (i + 2) == 0: return False i += 6 return True Then, our main.py file looks like this: Python from fastapi import FastAPI, HTTPException from is_prime import is_prime app = FastAPI() # Route to check if a number is a prime number @app.get("/prime/{number}") def check_if_prime(number: int): return is_prime(number) raise HTTPException(status_code=400, detail="Input invalid") if __name__ == "__main__": import uvicorn uvicorn.run(app, host="localhost", port=8000) That’s all there is to it. We can start our API locally (python main.py) and send some requests to try it out: Plain Text ~$ curl http://localhost:8000/prime/91 false ~$ curl http://localhost:8000/prime/97 true That looks pretty good. But we’d feel better with a unit test for the is_prime function. Let’s get to it. Step #1: Write Your Tests With pytest added to our Python dependencies, we’ll write a file called test_is_prime.py and put it in a subfolder called tests. We have a set of numbers that we’ll test to make sure our function determines correctly if they are prime or not. Here’s our test file: Python from is_prime import is_prime def test_1_is_not_prime(): assert not is_prime(1) def test_2_is_prime(): assert is_prime(2) def test_3_is_prime(): assert is_prime(3) def test_4_is_not_prime(): assert not is_prime(4) def test_5_is_prime(): assert is_prime(5) def test_991_is_prime(): assert is_prime(991) def test_993_is_not_prime(): assert not is_prime(993) def test_7873_is_prime(): assert is_prime(7873) def test_7802143_is_not_prime(): assert not is_prime(7802143) When we run pytest from the command line, here’s what we see: Python ~/project$ pytest =========================== test session starts =========================== platform linux -- Python 3.8.10, pytest-8.0.2, pluggy-1.4.0 rootdir: /home/michael/project/tests plugins: anyio-4.3.0 collected 9 items test_is_prime.py ......... [100%] ============================ 9 passed in 0.02s ============================ Our tests pass! It looks like is_prime is doing what it’s supposed to. Step #2: Deploy Your Heroku App It’s time to wire up Heroku. Assuming you have a Heroku account and you’ve installed the CLI, creating your app is going to go pretty quickly. Heroku will look in our project root folder for a file called requirements.txt, listing the Python dependencies our project has. This is what the file should look like: Plain Text fastapi==0.110.1 pydantic==2.7.0 uvicorn==0.29.0 pytest==8.0.2 Next, Heroku will look for a file called Procfile to determine how to start our Python application. Procfile should look like this: Plain Text web: uvicorn main:app --host=0.0.0.0 --port=${PORT} With those files in place, let’s create our app. Plain Text ~/project$ heroku login ~/project$ heroku apps:create is-it-prime That's it. Step #3: Push Your Code to Heroku Next, we push our project code to the git remote that the Heroku CLI set up when we created our app. Plain Text ~/project$ git push heroku main … remote: -----> Launching... remote: Released v3 remote: https://is-it-prime-2f2e4fe7adc1.herokuapp.com/ deployed to Heroku So, that’s done. Let’s check our API. Plain Text $ curl https://is-it-prime-2f2e4fe7adc1.herokuapp.com/prime/91 false $ curl https://is-it-prime-2f2e4fe7adc1.herokuapp.com/prime/7873 true $ curl https://is-it-prime-2f2e4fe7adc1.herokuapp.com/prime/7802143 false It works! Step #4: Create a Heroku Pipeline To Use Heroku CI Now, we want to create a Heroku Pipeline with CI enabled so that we can run our tests. We create the pipeline (called is-it-prime-pipeline), adding the app we created above to the staging phase of the pipeline. Plain Text $ heroku pipelines:create \ --app=is-it-prime \ --stage=staging \ is-it-prime-pipeline Creating is-it-prime-pipeline pipeline... done Adding ⬢ is-it-prime to is-it-prime-pipeline pipeline as staging... done With our pipeline created, we want to connect it to a GitHub repo so that our actions on the repo (such as new pull requests or merges) can trigger events in our pipeline (like automatically running the test suite). Plain Text $ heroku pipelines:connect is-it-prime-pipeline -r capnMB/heroku-ci-demo Linking to repo... done As you can see, I’m connecting my pipeline to my GitHub repo. When something like a pull request or a merge occurs in my repo, it will trigger the Heroku CI to run the test suite. Next, we need to configure our test environment in an app.json manifest. Our file contents should look like this: Plain Text { "environments": { "test": { "formation": { "test": { "quantity": 1, "size": "standard-1x" } }, "scripts": { "test": "pytest" } } } } This manifest contains the script we would use to run through our test suite. It also specifies the dyno size we (standard-1x) would want to use for our test environment. We commit this file to our repo. Finally, in the web UI for Heroku, we navigate to the Tests page of our pipeline, and we click the Enable Heroku CI button. After enabling Heroku CI, here’s what we see: Step #5: Run Your Tests With Heroku CI Just to demonstrate it, we can manually trigger a run of our test suite using the CLI: Plain Text $ heroku ci:run --pipeline is-it-prime-pipeline … -----> Running test command `pytest`... ========================= test session starts ============================ platform linux -- Python 3.12.3, pytest-8.0.2, pluggy-1.4.0 rootdir: /app plugins: anyio-4.3.0 collected 9 items tests/test_is_prime.py ......... [100%] ============================ 9 passed in 0.03s ============================ How does the test run look in our browser? We navigate to our pipeline and click Tests. There, we see our first test run in the left-side nav. A closer inspection of our tests shows this: Awesome. Now, let’s push some new code to a branch in our repo and watch the tests run! We create a new branch (called new-test), adding another test case to test_is_prime.py. As soon as we push our branch to GitHub, here’s what we see: Heroku CI detects the pushed code and automates a new run of the test suite. Not too long after, we see the successful results: Heroku CI for the Win If you’re already using Heroku for your production environment — and you’re ready to go all in with DevOps — then using pipelines and Heroku CI may be the way to go. Rather than using different tools and platforms for building, testing, reviewing, staging, and releasing to production… I can consolidate all these pieces in a single Heroku Pipeline and get automated testing with every push to my repo.

By Michael Bogan CORE

I am an avid reader of technical books, specifically those focused on Cloud, DevOps, and Site Reliability Engineering (SRE). In this post, I will share a list of books that I believe are essential for anyone looking to start or advance their career in Cloud, DevOps, or SRE. These books will help you build a strong foundation in the top skills required in these fields. While this post focuses primarily on Amazon AWS for public cloud, I will also include a few vendor-neutral books. Note: This is my honest opinion, and I am not affiliated with any of these book authors or publishers. How Linux Works, 3rd Edition: What Every Superuser Should Know by Brian Ward and the Linux Command Line, 2nd Edition by William Shotts Learning Linux is the first step before acquiring any other skills in DevOps. These books are excellent for building a strong foundation in Linux internals and getting familiar with the Linux command line, which is essential for excelling in the DevOps space. Python Programming is the second most important skill after Linux for DevOps or SRE. I recommend starting with Python Cookbook: Recipes for Mastering Python 3. Begin with the basics, then move on to object-oriented concepts, databases, APIs, and scripting. Eventually, you should learn about MVC and other design patterns to build comprehensive products, not just scripts. As a production engineer, you will need to develop many infrastructure tools. Solutions Architect's Handbook — Third Edition and AWS Cookbook These books provide a comprehensive view of what an AWS engineer needs to know. They were particularly helpful in preparing for the AWS Solution Associate exam, covering topics such as MVC architecture, domain-driven design, container-based application architecture, cloud-native design patterns, and performance considerations. The AWS Cookbook is excellent for practical labs and contains useful topics like secure web content delivery, dynamic access with security groups, automated password rotation for RDS, and big data solutions. Terraform up and Running by Yevgeniy Brikman Terraform is a widely used infrastructure automation tool in DevOps. This book covers the basics, and intermediate topics like managing state files, creating reusable modules, Terraform syntax, and more. It also addresses the challenge of managing secrets and provides options for integrating with Docker and Kubernetes. The book concludes with strategies for managing Terraform code within a team. Continuous Integration and Deployment This skill is crucial for developers, DevOps, SRE, or any engineer involved in development or operations. Tools like Jenkins, GitLab, and GitHub Actions are commonly used. For Kubernetes environments, GitOps tools like Flux and ArgoCD are popular. I recommend Automating DevOps with GitLab CI/CD Pipelines by Christopher Cowell for those starting with CI/CD. Kubernetes This technology has been trending and growing rapidly. For theoretical knowledge, the Kubernetes documentation is sufficient, but for hands-on learning, I recommend The Kubernetes Bible: The Definitive Guide to Deploying and Managing Kubernetes Across Cloud and On-Prem Environments. Microservice development and deployment are on the rise. For AWS, small microservices-based products can be deployed with ECS but for large-scale products, Kubernetes is required. System Design This is a vendor-neutral skill. I recommend Designing Data-Intensive Applications by Martin Kleppmann to learn how to build reliable and scalable systems. Finally, I acknowledge that merely reading books won't make you an expert. You need to engage in extensive hands-on labs to excel in this field.

Twenty years ago, software was eating the world. Then around a decade ago, containers started eating software, heralded by the arrival of open source OCI standards. Suddenly, developers were able to package an application artifact in a container — sometimes all by themselves. And each container image could technically run anywhere — especially in cloud infrastructure. No more needing to buy VM licenses, look for Rackspace and spare servers, and no more contacting the IT Ops department to request provisioning. Unfortunately, the continuing journey of deploying containers throughout all enterprise IT estates hasn’t been all smooth sailing. Dev teams are confronted with an ever-increasing array of options for building and configuring multiple container images to support unique application requirements and different underlying flavors of commercial and open-source platforms. Even if a developer becomes an expert in docker build, and the team has enough daily time to keep track of changes across all components and dependencies, they are likely to see functional and security gaps appearing within their expanding container fleet. Fortunately, we are seeing a bright spot in the evolution of Cloud Native Buildpacks, an open-source implementation project pioneered at Heroku and adopted early at Pivotal, which is now under the wing of the CNCF. Paketo Buildpacks is an open-source implementation of Cloud Native Buildpacks currently owned by the Cloud Foundry Foundation. Paketo automatically compiles and encapsulates developer application code into containers. Here’s how this latest iteration of buildpacks supports several important developer preferences and development team initiatives. Open Source Interoperability Modern developers appreciate the ability to build on open-source technology whenever they can, but it’s not always that simple to decide between open-source solutions when vendors and end-user companies have already made architectural decisions and set standards. Even in an open-source-first shop, many aspects of the environment will be vendor-supported and offer opinionated stacks for specific delivery platforms. Developers love to utilize buildpacks because they allow them to focus on coding business logic, rather than the infinite combinations of deployment details. Dealing with both source and deployment variability is where Paketo differentiates itself from previous containerization approaches. So, it doesn’t matter whether the developer codes in Java, Go, nodeJS, or Python, Paketo can compile ready-to-run containers. And, it doesn’t matter which cloud IaaS resource or on-prem server it runs on. “I think we're seeing a lot more developers who have a custom platform with custom stacks, but they keep coming back to Paketo Buildpacks because they can actually plug them into a modular system,” said Forest Eckhardt, contributor and maintainer to the Paketo project. “I think that adoption is going well, a lot of the adopters that we see are DevOps or Operations leaders who are trying to deliver applications for their clients and external teams.” Platform Engineering With Policy Platform engineering practices give developers shared, self-service resources and environments for development work, reducing setup costs and time, and encouraging code, component, and configuration reuse. These common platform engineering environments can be offered within a self-service internal portal or an external partner development portal, sometimes accompanied by support from a platform team that curates and reviews all elements of the platform. If the shared team space has too many random uploads, developers will not be able to distinguish the relative utility or safety of various unvalidated container definitions and packages. Proper governance means giving developers the ability to build to spec — without having to slog through huge policy checklists. Buildpacks take much of the effort and risk out of the ‘last mile’ of platform engineering. Developers can simply bring their code, and Paketo Buildpacks detects the language, gathers dependencies, and builds a valid container image that fits within the chosen methodology and policies of the organization. DevOps-Speed Automation In addition to empowering developers with self-service resources, automating everything as much as possible is another core tenet of the DevOps movement. DevOps is usually represented as a continuous infinity loop, where each change the team promotes in the design/development/build/deploy lifecycle should be executed by automated processes, including production monitoring and feedback to drive the next software delivery cycle. Any manual intervention in the lifecycle should be looked at as the next potential constraint to be addressed. If developers are spending time setting up Dockerfiles and validating containers, that’s less time spent creating new functionality or debugging critical issues. Software Supply Chain Assurance Developers want to move fast, so they turn to existing code and infrastructure examples that are working for peers. Heaps of downloadable packages and source code snippets are ready to go on npm StackOverflow and DockerHub – many with millions of downloads and lots of upvotes and review stars. The advent of such public development resources and git-style repositories offers immense value for the software industry as a whole, but by nature, it also provides an ideal entry point for software supply chain (or SSC) attacks. Bad actors can insert malware and irresponsible ones can leave behind vulnerabilities. Scanning an application once exploits are baked in can be difficult. It’s about time the software industry started taking a page from other discrete industries like high-tech manufacturing and pharmaceuticals that rely on tight governance of their supply chains to maximize customer value with reduced risk. For instance, an automotive brand would want to know the provenance of every part that goes into a car they manufacture, a complete bill-of-materials (or BOM) including both its supplier history and its source material composition. Paketo Buildpacks automatically generates an SBOM (software bill-of-materials) during each build process, attached to the image, so there’s no need to rely on external scanning tools. The SBOM documents information about every component in the packaged application, for instance, that it was written with Go version 1.22.3, even though that original code was compiled. The Intellyx Take Various forms of system encapsulation routines have been around for years, well before Docker appeared. Hey, containers even existed on mainframes. But there’s something distinct about this current wave of containerization for a cloud-native world. Paketo Buildpacks provides application delivery teams with total flexibility in selecting their platforms and open-source components of choice, with automation and reproducibility. Developers can successfully build the same app, in the same way, thousands of times in a row, even if underlying components are updated. That’s why so many major development shops are moving toward modern buildpacks, and removing the black box around containerization — no matter what deployment platform and methodology they espouse. ©2024 Intellyx B.V. Intellyx is editorially responsible for this document. At the time of writing, Cloud Foundry Foundation is an Intellyx customer. No AI bots were used to write this content. Image source: Adobe Express AI.

Introduction to Secrets Management In the world of DevSecOps, where speed, agility, and security are paramount, managing secrets effectively is crucial. Secrets, such as passwords, API keys, tokens, and certificates, are sensitive pieces of information that, if exposed, can lead to severe security breaches. To mitigate these risks, organizations are turning to secret management solutions. These solutions help securely store, access, and manage secrets throughout the software development lifecycle, ensuring they are protected from unauthorized access and misuse. This article aims to provide an in-depth overview of secrets management in DevSecOps, covering key concepts, common challenges, best practices, and available tools. Security Risks in Secrets Management The lack of implementing secrets management poses several challenges. Primarily, your organization might already have numerous secrets stored across the codebase. Apart from the ongoing risk of exposure, keeping secrets within your code promotes other insecure practices such as reusing secrets, employing weak passwords, and neglecting to rotate or revoke secrets due to the extensive code modifications that would be needed. Here below are some of the risks highlighting the potential risks of improper secrets management: Data Breaches If secrets are not properly managed, they can be exposed, leading to unauthorized access and potential data breaches. Example Scenario A Software-as-a-Service (SaaS) company uses a popular CI/CD platform to automate its software development and deployment processes. As part of their DevSecOps practices, they store sensitive credentials, such as API keys and database passwords, in a secrets management tool integrated with their pipelines. Issue Unfortunately, the CI/CD platform they use experiences a security vulnerability that allows attackers to gain unauthorized access to the secrets management tool's API. This vulnerability goes undetected by the company's security monitoring systems. Consequence Attackers exploit the vulnerability and gain access to the secrets stored in the management tool. With these credentials, they are able to access the company's production systems and databases. They exfiltrate sensitive customer data, including personally identifiable information (PII) and financial records. Impact The data breach leads to significant financial losses for the company due to regulatory fines, legal fees, and loss of customer trust. Additionally, the company's reputation is tarnished, leading to a decrease in customer retention and potential business partnerships. Preventive Measures To prevent such data breaches, the company could have implemented the following preventive measures: Regularly auditing and monitoring access to the secrets management tool to detect unauthorized access. Implementing multi-factor authentication (MFA) for accessing the secrets management tool. Ensuring that the secrets management tool is regularly patched and updated to address any security vulnerabilities. Limiting access to secrets based on the principle of least privilege, ensuring that only authorized users and systems have access to sensitive credentials. Implementing strong encryption for storing secrets to mitigate the impact of unauthorized access. Conducting regular security assessments and penetration testing to identify and address potential security vulnerabilities in the CI/CD platform and associated tools. Credential Theft Attackers may steal secrets, such as API keys or passwords, to gain unauthorized access to systems or resources. Example Scenario A fintech startup uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as database passwords and API keys, in a secrets management tool integrated with their pipelines. Issue An attacker gains access to the company's internal network by exploiting a vulnerability in an outdated web server. Once inside the network, the attacker uses a variety of techniques, such as phishing and social engineering, to gain access to a developer's workstation. Consequence The attacker discovers that the developer has stored plaintext files containing sensitive credentials, including database passwords and API keys, on their desktop. The developer had mistakenly saved these files for convenience and had not securely stored them in the secrets management tool. Impact With access to the sensitive credentials, the attacker gains unauthorized access to the company's databases and other systems. They exfiltrate sensitive customer data, including financial records and personal information, leading to regulatory fines and damage to the company's reputation. Preventive Measures To prevent such credential theft incidents, the fintech startup could have implemented the following preventive measures: Educating developers and employees about the importance of securely storing credentials and the risks of leaving them in plaintext files. Implementing strict access controls and auditing mechanisms for accessing and managing secrets in the secrets management tool. Using encryption to store sensitive credentials in the secrets management tool, ensures that even if credentials are stolen, they cannot be easily used without decryption keys. Regularly rotating credentials and monitoring for unusual or unauthorized access patterns to detect potential credential theft incidents early. Misconfiguration Improperly configured secrets management systems can lead to accidental exposure of secrets. Example Scenario A healthcare organization uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as database passwords and API keys, in a secrets management tool integrated with their pipelines. Issue A developer inadvertently misconfigures the permissions on the secrets management tool, allowing unintended access to sensitive credentials. The misconfiguration occurs when the developer sets overly permissive access controls, granting access to a broader group of users than intended. Consequence An attacker discovers the misconfigured access controls and gains unauthorized access to the secrets management tool. With access to sensitive credentials, the attacker can now access the healthcare organization's databases and other systems, potentially leading to data breaches and privacy violations. Impact The healthcare organization suffers reputational damage and financial losses due to the data breach. They may also face regulatory fines for failing to protect sensitive information. Preventive Measures To prevent such misconfiguration incidents, the healthcare organization could have implemented the following preventive measures: Implementing least privilege access controls to ensure that only authorized users and systems have access to sensitive credentials. Regularly auditing and monitoring access to the secrets management tool to detect and remediate misconfigurations. Implementing automated checks and policies to enforce proper access controls and configurations for secrets management. Providing training and guidance to developers and administrators on best practices for securely configuring and managing access to secrets. Compliance Violations Failure to properly manage secrets can lead to violations of regulations such as GDPR, HIPAA, or PCI DSS. Example Scenario A financial services company uses a popular CI/CD platform to automate their software development and deployment processes. They store sensitive credentials, such as encryption keys and API tokens, in a secrets management tool integrated with their pipelines. Issue The financial services company fails to adhere to regulatory requirements for managing and protecting sensitive information. Specifically, they do not implement proper encryption for storing sensitive credentials and do not maintain proper access controls for managing secrets. Consequence Regulatory authorities conduct an audit of the company's security practices and discover compliance violations related to secrets management. The company is found to be non-compliant with regulations such as PCI DSS (Payment Card Industry Data Security Standard) and GDPR (General Data Protection Regulation). Impact The financial services company faces significant financial penalties for non-compliance with regulatory requirements. Additionally, the company's reputation is damaged, leading to a loss of customer trust and potential legal consequences. Preventive Measures To prevent such compliance violations, the financial services company could have implemented the following preventive measures: Implementing encryption for storing sensitive credentials in the secrets management tool to ensure compliance with data protection regulations. Implementing strict access controls and auditing mechanisms for managing and accessing secrets to prevent unauthorized access. Conducting regular compliance audits and assessments to identify and address any non-compliance issues related to secrets management. Lack of Accountability Without proper auditing and monitoring, it can be difficult to track who accessed or modified secrets, leading to a lack of accountability. Example Scenario A technology company uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as API keys and database passwords, in a secrets management tool integrated with their pipelines. Issue The company does not establish clear ownership and accountability for managing and protecting secrets. There is no designated individual or team responsible for ensuring that proper security practices are followed when storing and accessing secrets. Consequence Due to the lack of accountability, there is no oversight or monitoring of access to sensitive credentials. As a result, developers and administrators have unrestricted access to secrets, increasing the risk of unauthorized access and data breaches. Impact The lack of accountability leads to a data breach where sensitive credentials are exposed. The company faces financial losses due to regulatory fines, legal fees, and loss of customer trust. Additionally, the company's reputation is damaged, leading to a decrease in customer retention and potential business partnerships. Preventive Measures To prevent such lack of accountability incidents, the technology company could have implemented the following preventive measures: Designating a specific individual or team responsible for managing and protecting secrets, including implementing and enforcing security policies and procedures. Implementing access controls and auditing mechanisms to monitor and track access to secrets, ensuring that only authorized users have access. Providing regular training and awareness programs for employees on the importance of secrets management and security best practices. Conducting regular security audits and assessments to identify and address any gaps in secrets management practices. Operational Disruption If secrets are not available when needed, it can disrupt the operation of DevSecOps pipelines and applications. Example Scenario A financial institution uses a popular CI/CD platform to automate its software development and deployment processes. They store sensitive credentials, such as encryption keys and API tokens, in a secrets management tool integrated with their pipelines. Issue During a routine update to the secrets management tool, a misconfiguration occurs that causes the tool to become unresponsive. As a result, developers are unable to access the sensitive credentials needed to deploy new applications and services. Consequence The operational disruption leads to a delay in deploying critical updates and features, impacting the financial institution's ability to serve its customers effectively. The IT team is forced to troubleshoot the issue, leading to downtime and increased operational costs. Impact The operational disruption results in financial losses due to lost productivity and potential revenue. Additionally, the financial institution's reputation is damaged, leading to a loss of customer trust and potential business partnerships. Preventive Measures To prevent such operational disruptions, the financial institution could have implemented the following preventive measures: Implementing automated backups and disaster recovery procedures for the secrets management tool to quickly restore service in case of a failure. Conducting regular testing and monitoring of the secrets management tool to identify and address any performance issues or misconfigurations. Implementing a rollback plan to quickly revert to a previous version of the secrets management tool in case of a failed update or configuration change. Establishing clear communication channels and escalation procedures to quickly notify stakeholders and IT teams in case of operational disruption. Dependency on Third-Party Services Using third-party secrets management services can introduce dependencies and potential risks if the service becomes unavailable or compromised. Example Scenario A software development company uses a popular CI/CD platform to automate its software development and deployment processes. They rely on a third-party secrets management tool to store sensitive credentials, such as API keys and database passwords, used in their pipelines. Issue The third-party secrets management tool experiences a service outage due to a cyber attack on the service provider's infrastructure. As a result, the software development company is unable to access the sensitive credentials needed to deploy new applications and services. Consequence The dependency on the third-party secrets management tool leads to a delay in deploying critical updates and features, impacting the software development company's ability to deliver software on time. The IT team is forced to find alternative ways to manage and store sensitive credentials temporarily. Impact The dependency on the third-party secrets management tool results in financial losses due to lost productivity and potential revenue. Additionally, the software development company's reputation is damaged, leading to a loss of customer trust and potential business partnerships. Preventive Measures To prevent such dependencies on third-party services, the software development company could have implemented the following preventive measures: Implementing a backup plan for storing and managing sensitive credentials locally in case of a service outage or disruption. Diversifying the use of secrets management tools by using multiple tools or providers to reduce the impact of a single service outage. Conducting regular reviews and assessments of third-party service providers to ensure they meet security and reliability requirements. Implementing a contingency plan to quickly switch to an alternative secrets management tool or provider in case of a service outage or disruption. Insider Threats Malicious insiders may abuse their access to secrets for personal gain or to harm the organization. Example Scenario A technology company uses a popular CI/CD platform to automate their software development and deployment processes. They store sensitive credentials, such as API keys and database passwords, in a secrets management tool integrated with their pipelines. Issue An employee with privileged access to the secrets management tool decides to leave the company and maliciously steals sensitive credentials before leaving. The employee had legitimate access to the secrets management tool as part of their job responsibilities but chose to abuse that access for personal gain. Consequence The insider threat leads to the theft of sensitive credentials, which are then used by the former employee to gain unauthorized access to the company's systems and data. This unauthorized access can lead to data breaches, financial losses, and damage to the company's reputation. Impact The insider threat results in financial losses due to potential data breaches and the need to mitigate the impact of the stolen credentials. Additionally, the company's reputation is damaged, leading to a loss of customer trust and potential legal consequences. Preventive Measures To prevent insider threats involving secrets management, the technology company could have implemented the following preventive measures: Implementing strict access controls and least privilege principles to limit the access of employees to sensitive credentials based on their job responsibilities. Conducting regular audits and monitoring of access to the secrets management tool to detect and prevent unauthorized access. Providing regular training and awareness programs for employees on the importance of data security and the risks of insider threats. Implementing behavioral analytics and anomaly detection mechanisms to identify and respond to suspicious behavior or activities involving sensitive credentials. Best Practices for Secrets Management Here are some best practices for secrets management in DevSecOps pipelines: Use a dedicated secrets management tool: Utilize a specialized tool or service designed for securely storing and managing secrets. Encrypt secrets at rest and in transit: Ensure that secrets are encrypted both when stored and when transmitted over the network. Use strong access controls: Implement strict access controls to limit who can access secrets and what they can do with them. Regularly rotate secrets: Regularly rotate secrets (e.g., passwords, API keys) to minimize the impact of potential compromise. Avoid hardcoding secrets: Never hardcode secrets in your code or configuration files. Use environment variables or a secrets management tool instead. Use environment-specific secrets: Use different secrets for different environments (e.g., development, staging, production) to minimize the impact of a compromised secret. Monitor and audit access: Monitor and audit access to secrets to detect and respond to unauthorized access attempts. Automate secrets retrieval: Automate the retrieval of secrets in your CI/CD pipelines to reduce manual intervention and the risk of exposure. Regularly review and update policies: Regularly review and update your secrets management policies and procedures to ensure they are up-to-date and effective. Educate and train employees: Educate and train employees on the importance of secrets management and best practices for handling secrets securely. Use-Cases of Secrets Management For Different Tools Here are the common use cases for different tools of secrets management: IBM Cloud Secrets Manager Securely storing and managing API keys Managing database credentials Storing encryption keys Managing certificates Integrating with CI/CD pipelines Compliance and audit requirements by providing centralized management and auditing of secrets usage. Ability to dynamically generate and rotate secrets HashiCorp Vault Centralized secrets management for distributed systems Dynamic secrets generation and management Encryption and access controls for secrets Secrets rotation for various types of secrets AWS Secrets Manager Securely store and manage AWS credentials Securely store and manage other types of secrets used in AWS services Integration with AWS services for seamless access to secrets Automatic secrets rotation for supported AWS services Azure Key Vault Centralized secrets management for Azure applications Securely store and manage secrets, keys, and certificates Encryption and access policies for secrets Automated secrets rotation for keys, secrets, and certificates CyberArk Conjur Secrets management and privileged access management Secrets retrieval via REST API for integration with CI/CD pipelines Secrets versioning and access controls Automated secrets rotation using rotation policies and scheduled tasks Google Cloud Secret Manager Centralized secrets management for Google Cloud applications Securely store and manage secrets, API keys, and certificates Encryption at rest and in transit for secrets Automated and manual secrets rotation with integration with Google Cloud Functions These tools cater to different cloud environments and offer various features for securely managing and rotating secrets based on specific requirements and use cases. Implement Secrets Management in DevSecOps Pipelines Understanding CI/CD in DevSecOps CI/CD in DevSecOps involves automating the build, test, and deployment processes while integrating security practices throughout the pipeline to deliver secure and high-quality software rapidly. Continuous Integration (CI) CI is the practice of automatically building and testing code changes whenever a developer commits code to the version control system (e.g., Git). The goal is to quickly detect and fix integration errors. Continuous Delivery (CD) CD extends CI by automating the process of deploying code changes to testing, staging, and production environments. With CD, every code change that passes the automated tests can potentially be deployed to production. Continuous Deployment (CD) CD goes one step further than continuous delivery by automatically deploying every code change that passes the automated tests to production. This requires a high level of automation and confidence in the automated tests. Continuous Compliance (CC) CC refers to the practice of integrating compliance checks and controls into the automated CI/CD pipeline. It ensures that software deployments comply with relevant regulations, standards, and internal policies throughout the development lifecycle. DevSecOps DevSecOps integrates security practices into the CI/CD pipeline, ensuring that security is built into the software development process from the beginning. This includes performing security testing (e.g., static code analysis, dynamic application security testing) as part of the pipeline and managing secrets securely. The following picture depicts the DevSecOps lifecycles: Picture courtesy Implement Secrets Management Into DevSecOps Pipelines Implementing secrets management into DevSecOps pipelines involves securely handling and storing sensitive information such as API keys, passwords, and certificates. Here's a step-by-step guide to implementing secrets management in DevSecOps pipelines: Select a Secrets Management Solution Choose a secrets management tool that aligns with your organization's security requirements and integrates well with your existing DevSecOps tools and workflows. Identify Secrets Identify the secrets that need to be managed, such as database credentials, API keys, encryption keys, and certificates. Store Secrets Securely Use the selected secrets management tool to securely store secrets. Ensure that secrets are encrypted at rest and in transit and that access controls are in place to restrict who can access them. Integrate Secrets Management into CI/CD Pipelines Update your CI/CD pipeline scripts and configurations to integrate with the secrets management tool. Use the tool's APIs or SDKs to retrieve secrets securely during the pipeline execution. Implement Access Controls Implement strict access controls to ensure that only authorized users and systems can access secrets. Use role-based access control (RBAC) to manage permissions. Rotate Secrets Regularly Regularly rotate secrets to minimize the impact of potential compromise. Automate the rotation process as much as possible to ensure consistency and security. Monitor and Audit Access Monitor and audit access to secrets to detect and respond to unauthorized access attempts. Use logging and monitoring tools to track access and usage. Best Practices for Secrets Management Into DevSecOps Pipelines Implementing secrets management in DevSecOps pipelines requires careful consideration to ensure security and efficiency. Here are some best practices: Use a secrets management tool: Utilize a dedicated to store and manage secrets securely. Encrypt secrets: Encrypt secrets both at rest and in transit to protect them from unauthorized access. Avoid hardcoding secrets: Never hardcode secrets in your code or configuration files. Use environment variables or secrets management tools to inject secrets into your CI/CD pipelines. Rotate secrets: Implement a secrets rotation policy to regularly rotate secrets, such as passwords and API keys. Automate the rotation process wherever possible to reduce the risk of human error. Implement access controls: Use role-based access controls (RBAC) to restrict access to secrets based on the principle of least privilege. Monitor and audit access: Enable logging and monitoring to track access to secrets and detect any unauthorized access attempts. Automate secrets retrieval: Automate the retrieval of secrets in your CI/CD pipelines to reduce manual intervention and improve security. Use secrets injection: Use tools or libraries that support secrets injection (e.g., Kubernetes secrets, Docker secrets) to securely inject secrets into your application during deployment. Conclusion Secrets management is a critical aspect of DevSecOps that cannot be overlooked. By implementing best practices such as using dedicated secrets management tools, encrypting secrets, and implementing access controls, organizations can significantly enhance the security of their software development and deployment pipelines. Effective secrets management not only protects sensitive information but also helps in maintaining compliance with regulatory requirements. As DevSecOps continues to evolve, it is essential for organizations to prioritize secrets management as a fundamental part of their security strategy.

By Josephine Eskaline Joyce CORE

Ansible is one of the fastest-growing Infrastructure as Code (IaC) and automation tools in the world. Many of us use Ansible for Day 1 and Day 2 operations. One of the best analogies to understand the phases/stages/operations is defined on RedHat's website: "Imagine you're moving into a house. If Day 1 operations are moving into the house (installation), Day 2 operations are the 'housekeeping' stage of a software’s life cycle." Simply put, in a software lifecycle: Day 0: Design/planning phase - This phase involves preparation, initial planning, brainstorming, and preparing for the project. Typical activities in this phase are defining the scope, gathering requirements, assembling the development team, and setting up the development environments. For example, the team discusses the CI/CD platform to integrate the project with, the strategy for project management, etc. Day 1: Development/deployment phase - This phase marks the actual development activities such as coding, building features, and implementation based on the requirements gathered in the planning phase. Additionally, testing will begin to ensure early detection of issues (in development lingo, "bugs"). Day 2: Maintenance phase - This phase in which your project/software goes live and you keep a tap on the health of the project. You may need to patch or update the software and file feature requests/issues based on user feedback for your development team to work on. This is the phase where monitoring and logging (observability) play a crucial role. Ansible is an open-source tool written in Python and uses YAML to define the desired state of configuration. Ansible is used for configuration management, application deployment, and orchestration. It simplifies the process of managing and deploying software across multiple servers, making it one of the essential tools for system administrators, developers, and IT operations teams. With AI, generating Ansible code has become simpler and more efficient. Check out the following article to learn how Ansible is bringing AI tools to your Integrated Development Environment: "Automation, Ansible, AI." RedHat Ansible Lightspeed with IBM Watsonx code assistant At its core, Ansible employs a simple, agentless architecture, relying on SSH to connect to remote servers and execute tasks. This eliminates the need for installing any additional software or agents on target machines, resulting in a lightweight and efficient automation solution. Key Features of Ansible Here is a list of key features that Ansible offers: Infrastructure as Code (IaC) Ansible allows you to define your infrastructure and configuration requirements in code, enabling you to version control, share, and replicate environments with ease. For example, say you plan to move your on-premises application to a cloud platform. Instead of provisioning the cloud services and installing the dependencies manually, you can define the required cloud services and dependencies for your application like compute, storage, networking, security, etc., in a configuration file. That desired state is taken care of by Ansible as an Infrastructure as Code tool. In this way, setting up your development, test, staging, and production environments will easily avoid repetition. Playbooks Ansible playbooks are written in YAML format and define a series of tasks to be executed on remote hosts. Playbooks offer a clear, human-readable way to describe complex automation workflows. Using playbooks, you define the required dependencies and desired state for your application. Modules Ansible provides a vast collection of modules for managing various aspects of systems, networks, cloud services, and applications. Modules are idempotent, meaning they ensure that the desired state of the system is achieved regardless of its current state. For example, ansible.bultin.command is a module that helps you to execute commands on a remote machine. You can either use modules that are built in, like dnf, yum, etc., as part of Ansible Core, or you can develop your own modules in Ansible. To further understand the Ansible modules, check out this topic on RedHat. Inventory Management Ansible uses an inventory file to define the hosts it manages. This inventory can be static or dynamic, allowing for flexible configuration management across different environments. An inventory file (.ini or .yaml) is a list of hosts or nodes on which you install, configure, or set up a software, add a user, or change the permissions of a folder, etc. Refer to how to build an inventory for best practices. Roles Roles in Ansible provide a way to organize and reuse tasks, variables, and handlers. They promote code reusability and help maintain clean and modular playbooks. You can group tasks that are repetitive as a role to reuse or share with others. One good example is pinging a remote server, you can move the tasks, variables, etc., under a role to reuse. Below is an example of a role directory structure with eight main standard directories. You will learn about a tool to generate this defined structure in the next section of this article. Shell roles/ common/ # this hierarchy represents a "role" tasks/ # main.yml # <-- tasks file can include smaller files if warranted handlers/ # main.yml # <-- handlers file templates/ # <-- files for use with the template resource ntp.conf.j2 # <------- templates end in .j2 files/ # bar.txt # <-- files for use with the copy resource foo.sh # <-- script files for use with the script resource vars/ # main.yml # <-- variables associated with this role defaults/ # main.yml # <-- default lower priority variables for this role meta/ # main.yml # <-- role dependencies library/ # roles can also include custom modules module_utils/ # roles can also include custom module_utils lookup_plugins/ # or other types of plugins, like lookup in this case webtier/ # same kind of structure as "common" was above, done for the webtier role monitoring/ # "" fooapp/ Beyond Automation Ansible finds applications in several areas. Configuration management: Ansible simplifies the management of configuration files, packages, services, and users across diverse IT infrastructures. Application deployment: Ansible streamlines the deployment of applications by automating tasks such as software installation, configuration, and version control. Continuous Integration/Continuous Deployment (CI/CD): Ansible integrates seamlessly with CI/CD pipelines, enabling automated testing, deployment, and rollback of applications. Orchestration: Ansible orchestrates complex workflows involving multiple servers, networks, and cloud services, ensuring seamless coordination and execution of tasks. Security automation: Ansible helps enforce security policies, perform security audits, and automate compliance checks across IT environments. Cloud provisioning: Ansible's cloud modules facilitate the provisioning and management of cloud resources on platforms like IBM Cloud, AWS, Azure, Google Cloud, and OpenStack. The list is not exhaustive, so only a subset of applications is included above. Ansible can act as a security compliance manager by enforcing security policies and compliance standards across infrastructure and applications through patch management, configuration hardening, and vulnerability remediation. Additionally, Ansible can assist in setting up monitoring and logging, automating disaster recovery procedures (backup and restore processes, failovers, etc.,), and integrating with a wide range of tools and services, such as version control systems, issue trackers, ticketing systems, and configuration databases, to create end-to-end automation workflows. Tool and Project Ecosystem Ansible provides a wide range of tools and programs like Ansible-lint, Molecule for testing Ansible plays and roles, yamllint, etc. Here are additional tools that are not mentioned in the Ansible docs: Ansible Generator: Creates the necessary folder/directory structure; comes in handy when you create Ansible roles AWX: Provides a web-based user interface, REST API, and task engine built on top of Ansible; Comes with an awx-operator if you are planning to set up on a container orchestration platform like RedHat OpenShift Ansible VS Code extension by Red Hat: Syntax highlighting, validation, auto-completion, auto-closing Jinja expressions ("{{ my_variable }") etc. The Ansible ecosystem is very wide. This article gives you just a glimpse of the huge set of tools and frameworks. You can find the projects in the Ansible ecosystem on Ansible docs. Challenges With Ansible Every tool or product comes with its own challenges. Learning curve: One of the major challenges with Ansible is the learning curve. Mastering the features and best practices can be time-consuming, especially for users new to infrastructure automation or configuration. Complexity: Initially, understanding the terminology, folder structure, and hierarchy challenges the user. Terms like inventory, modules, plugins, tasks, playbooks, etc., are hard to understand in the beginning. As the number of nodes/hosts increases, the complexity of managing the playbooks, and orchestrating increases. Troubleshooting and error handling: For beginners, troubleshooting errors and debugging playbooks can be challenging. Especially, understanding error messages and identifying the root cause of failures requires familiarity with Ansible's syntax and modules, etc. Conclusion In this article, you learned that Ansible as an open-source tool can be used not only for automation but also for configuration, deployment, and security enablement. You also understood the features, and challenges and learned about the tools Ansible and the community offers. Ansible will become your go-to Infrastructure as Code tool once you pass the initial learning curve. To overcome the initial complexity, here's a GitHub repository with Ansible YAML code snippets to start with. Happy learning. If you like this article, please like and share it with your network.

By Vidyasagar (Sarath Chandra) Machupalli FBCS CORE

Hello! My name is Roman Burdiuzha. I am a Cloud Architect, Co-Founder, and CTO at Gart Solutions. I have been working in the IT industry for 15 years, a significant part of which has been in management positions. Today I will tell you how I find specialists for my DevSecOps and AppSec teams, what I pay attention to, and how I communicate with job seekers who try to embellish their own achievements during interviews. Starting Point I may surprise some of you, but first of all, I look for employees not on job boards, but in communities, in general chats for IT specialists, and through acquaintances. This way you can find a person with already existing recommendations and make a basic assessment of how suitable he is for you. Not by his resume, but by his real reputation. And you can already know him because you are spinning in the same community. Building the Ideal DevSecOps and AppSec Team: My Hiring Criteria There are general chats in my city (and not only) for IT specialists, where you can simply write: "Guys, hello, I'm doing this and I'm looking for cool specialists to work with me." Then I send the requirements that are currently relevant to me. If all this is not possible, I use the classic options with job boards. Before inviting for an interview, I first pay attention to the following points from the resume and recommendations. Programming Experience I am sure that any security professional in DevSecOps and AppSec must know the code. Ideally, all security professionals should grow out of programmers. You may disagree with me, but DevSecOps and AppSec specialists should work with code to one degree or another, be it some YAML manifests, JSON, various scripts, or just a classic application written in Java, Go, and so on. It is very wrong when a security professional does not know the language in which he is looking for vulnerabilities. You can't look at one line that the scanner highlighted and say: "Yes, indeed, this line is exploitable in this case, or it's false." You need to know the whole project and its structure. If you are not a programmer, you simply will not understand this code. Taking Initiative I want my future employees to be proactive — I mean people who work hard enough, do big tasks, have ambitions, want to achieve, and spend a lot of time on specific tasks. I support people's desire to develop in their field, to advance in the community, and to look for interesting tasks and projects for themselves, including outside of work. And if the resume indicates the corresponding points, I will definitely highlight it as a plus. Work-Life Balance I also pay a lot of attention to this point and I always talk about it during the interview. The presence of hobbies and interests in a person indicates his ability to switch from work to something else, his versatility and not being fixated on one job. It doesn't have to be about active sports, hiking, walking, etc. The main thing is that a person's life has not only work but also life itself. This means that he will not burn out in a couple of years of non-stop work. The ability to rest and be distracted acts as a guarantee of long-term employment relationships. In my experience, there have only been a couple of cases when employees had only work in their lives and nothing more. But I consider them to be unique people. They have been working in this rhythm for a long time, do not burn out, and do not fall into depression. You need to have a certain stamina and character for this. But in 99% of cases, overwork and inability to rest are a guaranteed departure and burnout of the employee in 2-3 years. At the moment, he can do a lot, but I don't need to change people like gloves every couple of years. Education I graduated from postgraduate studies myself, and I think this is more a plus than a minus. You should check the availability of certificates and diplomas of education specified in the resume. Confirmation of qualifications through certificates can indicate the veracity of the declared competencies. It is not easy to study for five years, but at the same time, when you study, you are forced to think in the right direction, analyze complex situations, and develop something that has scientific novelty at present and can be used in the future with benefit for people. And here, in principle, it is the same: you combine common ideas with colleagues and create, for example, progressive DevOps, which allows you to further help people; in particular, in the security of the banking sector. References and Recommendations I ask the applicant to provide contacts of previous employers or colleagues who can give recommendations on his work. If a person worked in the field of information security, then there are usually mutual acquaintances with whom I also communicate and who can confirm his qualifications. What I Look for in an Interview Unfortunately, not all aspects can be clarified at the stage of reading the resume. The applicant may hide some things in order to present themselves in a more favorable light, but more often it is simply impossible to take into account all the points needed by the employer when compiling a resume. Through leading questions in a conversation with the applicant and his stories from previous jobs, I find out if the potential employee has the qualities listed below. Ability To Read It sounds funny, but in fact, it is not such a common quality. A person who can read and analyze can solve almost any problem. I am absolutely convinced of this because I have gone through it myself more than once. Now I try to look for information from many sources, I actively use the same ChatGPT and other similar services just to speed up the work. That is, the more information I push through myself, the more tasks I will solve, and, accordingly, I will be more successful. Sometimes I ask the candidate to find a solution to a complex problem online and provide him with material for analysis, I look at how quickly he can read and conduct a qualitative analysis of the provided article. Analytical Mind There are two processes: decomposition and composition. Programmers usually use the second part. They conduct compositional analysis, that is, they assemble some artifact from the code that is needed for further work. An information security analyst or security specialist uses decomposition. That is, on the contrary, it disassembles the artifact into its components and looks for vulnerabilities. If a programmer creates, then a security specialist disassembles. An analytical mind is needed in the part that is related to how someone else's code works. In the 90s, for example, we talked about disassembling if the code was written in assembler. That is, you have a binary file, and you need to understand how it works. And if you do not analyze all entry and exit points, all processes, and functions that the programmer has developed in this code, then you cannot be sure that the program works as intended. There can be many pitfalls and logical things related to the correct or incorrect operation of the program. For example, there is a function that can be passed a certain amount of data. The programmer can consider this function as some input numerical data that can be passed to it, or this data can be limited by some sequence or length. For example, we enter the card number. It seems like the card number has a certain length. But, at the same time, any analyst and you should understand that instead of a number there can be letters or special characters, and the length may not be the same as the programmer came up with. This also needs to be checked, and all hypotheses need to be analyzed, to look at everything much wider than what is embedded in the business logic and thinking of the programmer who wrote it all. How do you understand that the candidate has an analytical mind? All this is easily clarified at the stage of "talking" with the candidate. You can simply ask questions like: "There is a data sample for process X, which consists of 1000 parameters. You need to determine the most important 30. The analysis task will be solved by 3 groups of analysts. How will you divide these parameters to obtain high efficiency and reliability of the analysis?" Experience Working in a Critical Situation It is desirable that the applicant has experience working in a crunch; for example, if he worked with servers with some kind of large critical load and was on duty. Usually, these are night shifts, evening shifts, on a weekend, when you have to urgently raise and restore something. Such people are very valuable. They really know how to work and have personally gone through different "pains." They are ready to put out fires with you and, most importantly, are highly likely to be more careful than others. I worked for a company that had a lot of students without experience. They very often broke a lot of things, and after that, it was necessary to raise all this. This is, of course, partly a consequence of mentoring. You have to help, develop, and turn students into specialists, but this does not negate the "pain" of correcting mistakes. And until you go through all this with them, they do not become cool. If a person participated in these processes and had the strength and ability to raise and correct, this is very cool. You need to select and take such people for yourself because they clearly know how to work. How To Avoid Being Fooled by Job Seekers Job seekers may overstate their achievements, but this is fairly easy to verify. If a person has the necessary experience, you need to ask them practical questions that are difficult to answer without real experience. For example, I ask about the implementation of a particular practice from DevSecOps, that is, what orchestrator he worked in. In a few words, the applicant should write, for example, a job in which it was all performed, and what tool he used. You can even suggest some keys from this vulnerability scanner and ask what keys and in what aspect you would use to make everything work. Only a specialist who has worked with this can answer these questions. In my opinion, this is the best way to check a person. That is, you need to give small practical tasks that can be solved quickly. It happens that not all applicants have worked and are working with the same as me, and they may have more experience and knowledge. Then it makes sense to find some common questions and points of contact with which we worked together. For example, just list 20 things from the field of information security and ask what the applicant is familiar with, find common points of interest, and then go through them in detail. When an applicant brags about having developments in interviews, it is also better to ask specific questions. If a person tells without hesitation what he has implemented, you can additionally ask him some small details about each item and direction. For example, how did you implement SAST verification, and with what tools? If he tells in detail and, possibly, with some additional nuances related to the settings of a particular scanner, and this fits into the general concept, then the person lived by this and used what he is talking about. Wrapping Up These are all the points that I pay attention to when looking for new people. I hope this information will be useful both for my Team Lead colleagues and for job seekers who will know what qualities they need to develop to successfully pass the interview.