This section lists a set of patterns and anti-patterns linked to CI in a logical order: from source code management to packaging. All examples provided are based on known free open-source tools to test their integration in any context.
Version Control
Continuous integration begins before we even talk about automation or process. The implementation of CI starts first at the source code manager level.
Build on Every Commit
The main objective of any automated process is to facilitate application management in order to accelerate its production and maintenance. One of the most important metrics in this type of process is deployment frequency. This measure is crucial for a successful DevOps transformation and, therefore, requires a clear definition and supervision.
It is important to define the origin point, or points, of a new build because they often have different requirements and will determine the production frequency and, subsequently, the velocity of development — two metrics that are extremely important for a company's competitiveness today.
Examples of parameters that could be considered the origin of a build:
- A new commit often means a very fast production pace because there is no need for approval.
- A precise pattern in the validation message creates a rhythm that can be maintained but is nevertheless controlled by the development team because they always have the choice to start a build at any time without a review — though not necessarily upon each commit.
- A pull-request controls the speed of releases because one or more levels of review and approval are required.
Pattern: Build on every change, commit, branch, merge, and pull request
Anti-Pattern: Build once per sprint, per week, or per day; cherry pick commits
Define Conventions
The definition of standards plays an important role in the continuous integration chain, even if their role is often underestimated. Now, there are several recognized conventions in the DevOps world that make it possible to facilitate the understanding of an application's source code, lifecycle, and level of advancement, which can also help onboard new people to the project. Defining conventions, therefore, has major impacts both on the individual or team, and at the level of automated processes. Indeed, following standards facilitates and even minimizes the integration work required by DevOps teams.
Here are some examples of recommended conventions:
Example of a Git command to follow the conventions listed above:
git checkout -b AB-0001
git add -A
git commit -m "feat(scope): message"
git push
git tag 1.0.1
Pattern: Use the ticket number as a branch naming convention, add useful information to commit messages, and standardize the version for all applications
Anti-Pattern: Use nonrelevant branch naming, add meaningless commit messages, and use different version conventions for one or multiple applications
Shift Left
DevOps has significantly changed the day-to-day work of development and operations teams, enabling them to work together better and share knowledge. This transformation was also applied to security teams and gave rise to what's known as DevSecOps. This reconciliation of teams has led to a new approach called "shift left." Put simply, the shift-left principle consists of testing and securing applications — and therefore the source code — as early as possible in the integration stage, notably by adding automated processes to each commit, merge, or pull request.
Here are some recommended approaches to shift application testing and security left:
- Configure pre-commit tasks to lint and format the source code
- Configure a pre-commit task to control file structure (like the definition of a YAML file)
- Run a test on the core functionality of an application before merging code to the main branch
Pattern: Use pre and post actions on commits, merges, and pull requests
Anti-Pattern: Test source code after packaging or deploying it
Build
The build phase is the most important part of the continuous integration cycle. In this stage, code commits merge with security tests and validations but require several considerations to ensure apps are packaged properly.
Isolate Environments
A build automation tool can support multiple source codes at once. It is important to isolate each of their builds in order to:
- Avoid installing dependencies on a shared system
- Make the build environment portable and reproducible on any platform
- Give developers flexibility to update their pipeline without the need for the operations team
- Control the resources allocated to each build and, thus, control its budget and capital in a cloud environment
Pattern: Use a new, fresh isolated workspace to build the application and control the allocated resources to avoid impacting other builds
Anti-Pattern: Always use the same environment without identifying potential dependency issues; use all the resources available and potentially impact other builds
Implement Automated/Triggered Builds
This phase is the body of a continuous integration chain; its purpose is to create a deliverable (a compiled file, a container, a compressed file, etc.) that can then be deployed in production. Avoiding any manual action to promote automation of the deliverable's construction allows accelerated frequency of builds. It is therefore recommended to rely on webhooks to automatically start a pipeline upon each commit, each merge, or any other actions performed at the source code manager level.
It is also recommended to periodically start pipelines for each application currently in production for one simple reason: Code deployed in production can become vulnerable over time. New vulnerabilities may be introduced, and it is important to audit the entire architecture regularly to detect these anomalies and correct them as soon as possible to guarantee data security. Building a source code that is no longer updated but still in production can prevent this.
Pattern: Automatically release and deploy a new version on every commit, branch, merge, or pull request; test the build weekly to identify potential issues proactively instead of waiting for a code update
Anti-Pattern: Start a build manually at the end of a sprint or once a week; wait for a new ticket to build an application
Manage Hotfixes
A hotfix is generally defined as a patch to a live system due to a bug or vulnerability that meets a certain level of risk and severity. Normally, a hotfix is created as an urgent action against problems that need to be fixed immediately and outside of the normal git workflow. As part of a software development cycle, the development team should have a flexible definition of a hotfix and an internal method for determining what meets the needs for a hotfix.
When a critical bug in a production version must be resolved, a hotfix branch may be plugged off from the corresponding tag on the main branch that marks the production version. That way, the team members can continue working on the development branch while another person prepares a quick production fix.
Pattern: Deploy a hotfix as soon as possible; test the code in staging before moving it to production
Anti-Pattern: Schedule the deployment of a hotfix; test directly in production
Control the Source Code
Controlling the vulnerability of deliverables is a crucial point in a CI process. The DevSecOps methodology requires early integration of key validation points, such as identification and control of dependencies of each built application and container. It's the developer's responsibility to ensure that the application deployed in production doesn't involve a critical severity that could be the source of system security breach. To avoid this, it's important to audit the source code continuously and break the pipeline if a severity is identified. Here is an example of a command that can be used to audit and break a Node.js pipeline if a critical vulnerability is detected:
$ npm audit –audit-level=critical
Pattern: Ensure the deliverable is free of any CVE (Critical Vulnerabilities and Exposures); take action immediately when a severity is identified
Anti-Pattern: Wait until the infrastructure is audited to find dependency issues
Check for Sensitive Data
While DevOps has rightfully gained ground with many software development teams, security is often overlooked because it interferes with shipping functions. This paradigm, wherein DevOps conflicts with security needs, has led to weak software practices like storing application secrets in code. Embedding sensitive data control points in the CI process in order to intercept accidental propagation of any code containing a secret as soon as possible helps avoid and prevent its exposure and leaks.
A simple way to enforce rules is to shift security controls left as a pre-commit or as a step in the pipeline, where source code can be scanned and control points be added to break the pipeline if any sensitive data is found. Some types of sensitive data that any scanner should identify in source code include passwords, usernames, and emails.
Pattern: Check the source for sensitive data and break the pipeline if it is found; dissociate passwords from the source code
Anti-Pattern: Add passwords in the configuration file and release it with the source code
Control Source Code Quality
"Clean as you code" is a recognized development practice that involves automatic and continuous control of code quality by integrating quality rules, gates, and profiles into each pipeline. Effective code quality and security practices should become second nature and be integrated into the workflow to facilitate the maintenance and implementation of application features.
Code smells are often good indicators to measure the quality of an application's code and identify any code that could eventually lead to serious failures and kill an application's performance. Typical aspects of code smells are:
- Duplicate code
- Dead code
- Long methods or parameter list
Pattern: Lint and format code automatically to make it more readable; break the pipeline upon receiving bad quality reports
Anti-Pattern: Don't define standards to follow; release duplicated code
Automate Testing
The importance of testing is clear and even more pertinent in the continuous integration process. Indeed, being able to shorten the feedback loop and detect any potential issues in the code that's integrated into the main branch of development is a critical issue for software developers. Each change made to the codebase can potentially impact the platform's stability, which is why having a process for automated testing is vital.
The formulation of automating tests is generic and can be done at multiple levels. Both quality assurance best practices and the testing pyramid explain that unit tests should be the primary part of the testing process. They test individual components or functionalities to validate that they work as expected in isolated conditions. Other tests like integration tests, which ensure communication between components is working properly, are also a necessity in the continuous integration process. Finally, end-to-end tests confirm the application works flawlessly from start to finish, often reproducing an end-user scenario.
Pattern: Run a set of tests automatically on each build; run specific tests periodically
Anti-Pattern: Wait for the deployment of a package to run tests manually
Mock the Environment
Mocking means to create a fake version of an external or internal service that can replace the real one in order to test the source code faster and more reliably. Mocking is important for ensuring the portability and reproducibility of the CI pipeline.
Pattern: Run tests the same way on any platform (laptop, on-premises, cloud, container orchestration platform, etc.) to always have the same result with mocked data
Anti-Pattern: Run tests that will potentially fail based on the status of the infrastructure
Monitoring
Monitoring automated pipelines is one of the keys to a successful DevOps transformation. It is important to supervise and control the entire integration process in order to identify anomalies and improvement points as soon as possible. Neglecting this could have various costs at multiple levels, including:
- Architecture – Technical debt can multiply, which is also accelerated by poor integration.
- Team resources – Time needed to maintain an application, develop new features, correct and detect bugs, and monitor dependencies increases.
- Company – Simple anomalies in production, or even failures, can compromise SLAs, induce poor user feedback, and harm the company's reputation.
A way to mitigate these risks is to implement an observability platform to continuously measure the pipelines and their statuses, code quality, frequency of dependency issues, and more. Centralizing data and rendering it in a way that all stakeholders can understand is crucial for improving team collaboration and CI process adoption.
The following is a list of questions to answer to net the benefits of pipeline monitoring:
- Measure adoption – How many builds are started each day? By which team? For which application?
- Analyze trends – How many builds succeeded, failed, or were aborted?
- Identify improvements – Why is a build slower or faster than the previous day? How many new vulnerabilities have been identified compared to the last run?
Pattern: Measure the number of releases per day, the time to build each one, and pipeline status
Anti-Pattern: Ignore the performance of the CI pipeline
Communication
DevOps culture is established through collaboration between teams that traditionally work in silos. Development and operations teams do not normally have the same constraints, nor do they typically use the same processes, tools, project management styles, or methodologies. The challenge of a DevOps approach is to align both teams under a common goal and develop a unique team spirit among them. Some aspects of each team are and will remain different; therefore, transparent exchange of data and information is the cornerstone of effective communication and promotes effective feedback, a cohesive team, and the end of informal exchanges.
Pattern: Automatically notify all teams of a new release and tag or comment a ticket on a build status
Anti-Pattern: Wait for someone to ask if the new release is ready
Tools that enable effective communication will enhance cooperation and collaboration, establish efficient feedback channels, and improve the quality of shared information. Each build should automatically send the project's team members a notification with information such as:
- The name of the built Git branch
- The new version bumped by the process if the build succeeded
- The build status with, if possible, a visual representation (e.g., color-coding) to quickly identify a successful build from a failure
Pattern: Send a notification with meaningful information like the developer's name, project name, build branch, status, version, and a link to the build
Anti-Pattern: Communicate through different mediums and send varied formats of notifications without standards
The type of notification is also an important aspect of a CI process. Today, we have an impressive choice of mediums for communication to collaborate as one DevOps team. It is important to choose the right communication channel according to both the context of the notification and the company — and more specifically, the collaboration tool being used. Here are three straightforward ways to communicate:
- Chat – the most direct approach to notify a person, team, or all project team members at once
- Email – the traditional and slower way but allows archiving notifications
- Management ticket – a collaborative approach that allows automated task progress in a project management tool
Pattern: Implement a logical notification system allowing direct notification to relevant teams and higher-level monitoring to facilitate project management
Anti-Pattern: Don't send notifications and wait for someone to close a ticket or add a comment
Documentation
Documentation is the structured recording of information related to an IT ecosystem. This includes documentation of source code, configuration files, and internal standard operating procedures.
Document Workflows
A continuous integration process requires clear, useful documentation to support its adoption, facilitate effective implementation and use, and enable successful automation. This type of standardization between processes ensures that all members of an organization are aligned and work toward the same goals and outcomes.
Pattern: Document the integration architecture and pipelines so that all stakeholders understand the workflow and the required information to build a package
Anti-Pattern: Wait for questions and hope people will share information with their teammates; assume documentation can be postponed and schedule several training sessions to repeat information
Document Project Lifecycles
Documentation should be as important to a developer as any other facet of development; however, documenting code is unfortunately not often considered a high priority. The speed of development increases competition but taking the time to write an appropriate description of what has changed will save time, reduce risk, and support team member onboarding in the long run. Following conventions like the validation message format can help development teams document their projects and work automatically. Examples of automated documents that a pipeline can create:
- Update a
README file with the version of the artifact, developer's name, and a link to any tickets associated with the release
- Update a
CHANGELOG file with the artifact version and a list of commit messages from the previous version
- Update a
VERSION file with the last built version
Pattern: Implement and track documentation for all projects to share and centralize meaningful information
Anti-Pattern: Keep all changes in separate files and encourage developers to find information in the source code manager's history
Dissociate Configuration From Code
Adding flexibility to the application's management strategy is essential in a world where dynamic platforms (e.g., for container orchestration) are becoming more prominent and widely used. Source code and configuration files are two distinct components and should have their own management strategy. Source code should work the same way in all environments because it is immutable. Configuration files are an external part of the application that matters only during execution and can be overridden before starting the application.
Dissociating configuration files and source code makes it easier to maintain and secure them for many reasons, including:
- Operations can update the contents of a configuration file without changing the source code of an application.
- The same artifact can be promoted without having to rebuild it.
- The application is easily portable into a new environment.
- Security teams can better audit the access to sensitive data.
Pattern: Manage, deduplicate, and version the application's configuration into a centralized configuration management tool
Anti-Pattern: Store an application configuration file by environment in the source code
Release Your Data
Databases are the cornerstones of all modern software projects; no project of any scale beyond a prototype can function without some form of a database. Continuous database integration is the integration of the database schema and logical changes in application development efforts. Applying the same principles of integration and deployment patterns to the database allows all database changes to flow through the pipeline of each software version, synchronized with the application code.
The main goal is to keep a release's code aligned with the schema of a database, which is essential when launching a new feature — and even more crucial during a rollback where retro compatibility must be ensured.
Pattern: Automatically release and update a database schema before creating a new artifact; ensure backward compatibility with the current version deployed to successfully roll back
Anti-Pattern: Manually manage changes in a database before starting a new compilation
