Shifting Left With Continuous Delivery

How can a software development organization improve quality and speed time-to-market? By shifting left.

Think for a moment about a traditional development “waterfall.” First requirements are gathered and codified. Then, programmers implement the requirements. Then, testing and documentation take over, and after they finish, the result is released to operations. In this framework, any “shifting left” moves a segment of the process closer to the beginning of the timeline, on the left. “Shift left” largely began with testing. The idea was that starting testing before programming ended — perhaps as early as it began — helps reduce risk, identify problems earlier and at lower cost, and improve communications between development and testing.

Continuous delivery (CD) starts with that same waterfall timeline and aims to improve it with a different but related emphasis on “shifting left.” In the original waterfall timeline, a programmer’s source code is typically built into an application, the application is staged to a test environment, the app is tested, and if all goes well, the staged application is eligible for promotion to production or release. Each of these steps involves considerable human involvement and delay. CD automates most or all of these steps. With CD, a programmer completes, submits, or commits a code change, then automated systems immediately compile and test the application. Specific enhancements are available to benefit consumers within minutes or hours rather than weeks or months.

Shift-left and CD yield a software pipeline that’s sometimes orders of magnitude faster, as the descriptions above hint. That’s not all, though; the methodologies and automations of shift left and CD tighten the links between programmers and customer experience so that it’s far easier to validate and correct features. Most defects introduced into source result in errors reported that same day. CD greatly reduces the frequency of needing to “debug” an error found “on the right” due to a source change “on the left” from weeks earlier. Shift left and CD make software development faster and more reliable.

Most of the processes actually involved in CD are familiar to experienced programmers and administrators. The topic has collected a number of specialized terms, though, to help identify or emphasize a particular perspective:

Agile

Computer developers and managers use “agile” to emphasize “customer satisfaction through early and continuous delivery of useful software … close, daily cooperation between business people and developers …” and related principles. The two quotes here are among the dozen of tenets of the original Agile Manifesto written in 2001. Note that “continuous delivery” appears right at the very beginning of the manifesto, and therefore of agile culture.

Several Refcardz are already available that explain various aspects of Agile development:

• Design Patterns
• Getting Started With Lean Software Development
• Software Configuration Management Patterns

CD

The initialism CD stands for either “continuous delivery” or “continuous deployment.” The practices remain novel enough that practitioners don’t yet use the labels consistently. Generally, though, “continuous delivery” means that all the steps from programmer-commit through integration tests are automated. “Continous deployment” does everything “continous delivery” does, plus automatically delivers the latest tested code changes to customers.

The understanding of the words automation, validation, and delivery has changed throughout the years. In 2001, continuous deployment might have implied a daily release to a customer. In 2018, it generally brings to mind that end-users see an update a few seconds or minutes after a programmer checks in new code.

CI

“Continuous integration” (CI) is the first step of CD. CI automatically builds or compiles a usable application from the latest code changes in a software repository and ensures no changes overlap with each other. Grady Booch first wrote about CI in 1991, when the emphasis was on consolidation of the output of an entire programming team into a single authoritative source image. Discussions of CI now generally focus on optimizations or automations valuable for large bodies of code with complex build procedures.

Containers

A container virtualizes an application at the level of the operating system. While Docker is the market leader, many other container technologies are in widespread use, including OpenVZ, Hyper-V, and so on.

DevOps

Traditional computing differentiates development from operations. Different people work in the two domains, and communication between them is limited.

DevOps aims to bring these perspectives together. If a problem turns up in production in a traditional model, operations has the responsibility for diagnosing and correcting it, often without help. DevOps aims for problems to turn up sooner, before they reach production, and, when production is the location of a symptom, programmers pitch in to find and fix it. Presumably, this leads to programmers with better perspective on what can go wrong in operations and operators with more understanding of how programmers put the applications together.

Docker

Docker is a Linux-based container implementation in widespread use.

Hello, World

Programmers often focus on a minimal example of a technique or tool under the label of a “Hello, world.” A model introductory program in the C language appeared in an influential book printed “Hello, world” to the screen, and halted. As over-simplified as that might seem, experience proved that even such a minimal program usefully illustrates important principles, and therefore is good practice for beginners.

Lightweight

Digital solutions that emphasize ease of use, minimalism, or simplicity are often called “lightweight.”" The contrast is with “heavy” technologies that give more capabilities at the cost of more restrictions, computing resources, or procedural investment.

SDLC

The software development lifecycle (SDLC) is the comprehensive process an organization uses to plan, budget, develop, validate, maintain, replace, enhance, support, and eventually retire specific software assets.

Shift Left

If we built houses like software programs, we wouldn’t tell the roofers how big the house is until after the electrical wiring had passed inspection, let alone give them a chance to purchase their materials.

Shifting left rearranges the work of different computing specialists so that they can do more in parallel and cooperation, rather than isolation.

“Shift left” aims to begin acceptance testing far earlier, and more nimbly: left-shifted testing is far more likely to yield insight that informs fundamental architectural improvements. Left-shifted testing also involves deeper and more frequent communication between specialists in testing, operations, and development.

While “shift left” was first applied to testing, it now applies with even more force to operations and other segments of the SDLC.

Waterfall

The sequential development methodology is labeled “waterfall.” First, all requirements for a project are gathered and written; then, the completed requirements are delivered to architects, who design a solution. Programmers implement the design until they believe the application exhibits all necessary functionality. After the completed application passes to the hands of testers, the testers identify all defects they can find. These handoffs continue through stages of correction, delivery, operation, and maintenance. Waterfall communication focuses on “deliverables,” with a minimum of interactions between teams or stages.

Shift left and CD cover so much territory that it’s impossible to exhibit a single meaningful minimal working example that applies to every organization. When working with a programming language like Java or a technology such as a messaging service, developers often refer to a small working instance as a “Hello, world.” Java’s standardization is good enough that a model “Hello, world” is meaningful whether the programmer uses Linux or Windows, and whether the ultimate target is an automotive controller or a telephone application.

CD isn’t yet well-standardized. Practitioners will write their own minimal examples with these pieces:

• A way to retrieve a source bundle into a staging environment.
• An automatic validation (such as a test suite) that evaluates the latest source bundle.
• Automation that either accepts the latest bundle as good enough for release or rejects it, notifies responsible parties, and reverts the current application to a previous known-good milestone.

Such a program might be expressed in bash, PowerShell, Python, JavaScript, or an even more exotic language. A workload automation tool such as Jenkins or Control-M can help structure the lower-level scripting.

Useful references for a first foray into CD include:

• Testing in Continuous Delivery: Shift Left"
• Shift Left to Reduce Failure
• 5 Tips for Shifting Left in Continuous Testing
• How High-Performing Teams “Shift Left” With Continuous Testing
• How to Build a Modern CI-CD Pipeline

The most time-consuming difficulties in shifting left are more about attitude and culture than technology. At their best, CD initiatives can revolutionize an organization’s software development: CD technologies make it trivial to “spin up” debugging instances and isolate thorny problems in minutes, rather than the weeks or longer that sometimes were routine in old-fashioned waterfalls. CD only succeeds, of course, when decision-makers and front-line workers have a stake in those improvements.

Shifting left, therefore, has at least as much to do with business practices as it does with specific scripts, cloud products, and configuration management tools. A company can chose Git as the protocol for its source repository; whether programmers fully participate in code reviews, whether programmers are careful that their changes don’t degrade the test results of other parts of the application, and whether enhancements cycle quickly to customers are different and nearly independent questions, though.

Shifting left isn’t just something testers do; whole teams cooperate in effective left shifts.

Modern CD thinking recognizes that at least the most advanced software organizations automate delivery and even deployment to a remarkable degree. A programmer in a well left-shifted environment can check in a change, have it fully validated and analyzed, and then provide value to end users in just a few seconds at best. All the tools necessary for advanced automation exist now.

Emphasis on quality aims more to prevent problems rather than the need to detect and correct them. Issues are resolved as close to the “front line” and as early as possible.

Current shift-left testing theory emphasizes integration or “end-to-end” tests. While unit tests are valuable, heroic efforts to mock services in order to achieve 100% source coverage are misplaced. It is far better to establish priorities for an application, automate tests that reflect typical usage, and invest in a wider range of validations than just unit tests. Static analysis, performance testing, user-experience testing, and source metrics all play a part in improvement of quality.

Containers are crucial. It’s certainly possible to automate the SDLC fully without containers; but containers have proved useful so many times that every organization needs a container strategy. Good container design impacts programming, testing, operations, documentation, and more.

Programmers increasingly report that they work with the expectation of a well-equipped infrastructure. Instead of painstakingly “stick-building” a personal desktop over a week or more, modern practitioners expect to leverage cloud facilities and container technologies to snap together a good CD pipeline in under a minute.

It’s not just traditional testing that shifts left. All sorts of work are being redefined as software development, including software-defined networking, datacenter-as-a-service, factory automation, Jobs-as-Code, software-defined business, and so on. As each domain emerges as a species of software, it adopts techniques to shift left. Applied to Jobs-as-Code, for example, shifting left results in earlier detection and resolution of operational performance problems.

CD tooling is complex enough to promote specialization. Git, Jenkins, Selenium, Lambda, and so on are dense with features. It can be a full-time job to keep up with just one of these; teamwork is the only hope of tackling them all.

Shifting left is first a conceptual challenge. It is a lively and often confusing topic because it covers so much territory. Computer teams in the 1950s began to mention the value of testing earlier in workflow, and CD is nearly twenty years old as a label for automation in the SDLC.

However, the technologies involved and the focus for implementers have changed considerably during that time. Standardization of environments and technologies remains low. On one side, many substantial legacy operations still need to migrate to a reliabghle source repository; they’re still taking “baby steps” toward CD. At the other end of the spectrum are teams that fully adopt and assume comprehensive cloud-based SDLC workflows. Any conversations about CD need to start with clarity about exactly what terms and ideas are involved.

Best practices in CD are starting to emerge, though, and the requirements of what improvements need to be made are becoming clearer. Specific tools still must better integrate security and performance best practices.

Containers are already prominent in CD work and will continue to become more important. Machine learning also has the potential to help with more and more of the complexities of software development.

Validation will become more comprehensive over time. CD will report not just the boolean result — “this changeset is (not) good enough to go into production” — but will measure finer-grained metrics about potential weaknesses, performance vulnerabilities, areas for future security improvements, and suggestions for practitioner training.

Finally, as DevOps becomes more routine and solves today’s problems, attention will shift to aligning digital solutions to other functional domains. Now, for instance, when marketing or finance departments want to analyze or experiment with a new initiative, they submit requirements to development, which eventually delivers an implementation. Shifting left has accelerated and otherwise improved that last step. To what extent will future technologies put marketing results directly into the hands of marketing, though, without the need for development expertise? To shrink the necessity for Development expertise in these situations all the way to zero will be the ultimate payoff of shifting left.