8 DevOps Experiences to Take Into Account in Your Next Project

Starting a DevOps culture isn’t an overnight process. Whatever stage you are in your DevOps journey can be a challenge. Silos, handoffs, lack of goal-alignment, Frankenstein systems with no documentation, and missing cross-alignment between teams are all common obstacles organizations face and must address. How these types of challenges are addressed is paramount to being off to a DevOps journey where you get to ride off into the sunset or a not so happy one where everyone falls off a virtual cliff.

In this article, 8 experts look at a big DevOps challenge they faced and how they overcame them. Sometimes to make a dent in your organization on your DevOps journey, sometimes all you need as Justin Albano notes is a laptop and a goal.

5 of our experts in this article, also share their stories in the video below.

You Have to Prove It

By Justin Albano (@justinmalbano), Software Engineer at Catalogic Software, Inc.

While Agile is an effective process, we cannot expect everyone to be convinced of its usefulness without results. It’s our obligation to prove that it works and show how it can directly help our teams, especially on teams that are suspicious of new technologies and processes.

One of my most daunting DevOps challenges was working on a particular project in the military contracting space during my previous job. Intelligence and military work is notorious for being antiquated and behind-the-time, but for good reason: Almost all of the technology adopted needs to be thoroughly researched and approved, which is great for security but frustrating when trying to facilitate improvements in release times.

The product we were developing was brand new and I knew this was an opportunity to introduce Continuous Delivery (CD) into a world that was very leery of it. Since I did not have a server for builds or deployments, I used my own laptop—the same one I was writing the code on. Each time the code was checked in, a daemon running on my machine would check it out, build it through a pipeline I created, and store the binary on my machine. Each time a colleague asked for a binary to use, I could instantly provide him or her with one just by grabbing the result of the last pipeline build. It wasn’t the textbook CD approach and I wasn’t an expert, but it worked and it started to drive down the release time by orders of magnitude.

It’s difficult for improvements like that to go unnoticed and colleagues became curious. I started fielding questions about the pipelines I created and how CD worked in general. Not only released times improved, but innovative CD practices were gaining traction. In the midst of concurrently living Andy Hunt and Dave Thomas’s Stone Soup and Boiled Frogs stories, I realized an invaluable lesson: A first shot at making an improvement in the process—i.e. from legacy to agile processes—doesn’t have to be perfect, it just has to make a big enough improvement and people will take notice. I didn’t have a rack of build servers; I didn’t have hours of time to devote to release engineering; I didn’t have an agile culture. But I did have a laptop and a goal, and that’s all it took to make a dent.

Up next for Justin Albano:
Next, I'm working on something on the fundamentals of Java or Spring.

Learning to Come Together With DevOps

By Alireza Chegini, Senior DevOps Engineer / Site Reliability Engineer at Five Degrees

For me, another meaning for DevOps is always having challenges because I do not remember a day without a challenge as a DevOps engineer.

A couple of years ago, at the beginning of one of my projects, there was software that was quite tricky to install, especially that I was new to the company. There was no single documentation for the installation, and if you even would follow the current document, you were not still able to get the software running. The software was a legacy, and its architecture was complicated because of having too many configurations. So, every time you would try, something failed, and it was an entirely new experience: These unexpected situations were so tiring and frustrating. All the steps were manual, and the order of execution was not clear. I was thinking at that time how hard this could be for clients who were using this software to install, maintain, and support it. In the company, only a few people were able to install the software with some efforts because they have been doing it for years, and they know everything about it, but that knowledge could not be found anywhere. That is why no one could do what those experts did. Everyone agreed that something was not right, but there was no common agreement on the nature of the problem. Then there was no chance to think about possible solutions. This was a signal that there was a need for DevOps culture.

It seems that in any software company, we can expect from developers to DevOps to have at least a minimum understanding of the software and its architecture, to be able to develop, maintain, and support. This might look very clear for you as a reader but is not that known to the employees in that company. I should have done something about this. I came up with an idea to create an installation manual for dummies. Microsoft documentation styles have always inspired me. Usually, there is no need to know anything upfront for doing tutorials, so you would follow it step by step, and you are done! This is the way I like to create a manual by explaining everything and simplifying the instructions. Therefore, we did start with a small team of two persons; we installed the software and, at the same time, recorded all steps we did in a word document. In this manual, we explained the installation per each component with all configurations that were needed to be done. The manual was explained in a specific order so that you would start from the beginning to the end. You would have the software installed and running. We also added some simple smoke tests to be used to verify the result of the installation. Then we asked other people to follow the document and install the software and give us feedback about anything that was not clear. So, we put this practice in a continuous improvement loop. Then every time we get any feedback, we apply the feedback on the document and give it to the next person to test. After a while, the document becomes mature and straightforward enough. Therefore, we finished this rocket science project! Later, this document was used as a first initial input for the first-ever automated pipeline. One takeaway from this story is we did not find anything new about the software installation, which has not been known before us, but we did collect all the required information from different places and organized them in a structured way, which was familiar for everyone.

I think the most challenging situation is when you are at the beginning of implementing DevOps culture in a company. On the one hand, this is a unique opportunity to start designing, implementing DevOps practices right away from scratch, but on the other hand, this becomes quite difficult when a company, employees, and your colleagues are not ready for that, and they hardly know what DevOps even means. For many people, DevOps means CI/CD, automation and other technical things but to me, DevOps is about the collective ability to understand problems; how we can agree on simplifying processes; how we can speed up processes; how we can increase the performance and the last and the most important for me is how we can share the knowledge between each other. So, this is how I see the DevOps, and I think all automation and any other DevOps practices are successful when team working is in place.

Up next for Alireza Chegini:
I’m working on a trend report related to application performance monitoring using machine learning

DevOps the Voice Within Alexa Platform

By Xavier Portilla Edo (@Xavidop), Technical Leader at Altran

Without a doubt, the most important challenge that I have had to face has been to apply an entire DevOps process to an Alexa skill. Not so much because of the software components itself, but also because of the new challenges that we face when testing applications of this type.

We are facing a new application paradigm where the main interaction is the voice, that is, we are not facing a simple web page or a REST API. We are facing something totally new and revolutionary that time ago we could only find in laboratories or in products in the alpha phase. I compare it with what we experienced in 2008 when we started developing applications for our Smartphones. How do we test it? How do we automate all this? How do we ensure optimum quality in this kind of software?

One of the big challenges is in the testing part. In these applications, there are tests that have never been carried out on conventional software (natural language understanding tests, end-to-end tests with voice).

Amazon is aware of all these challenges and that is why it has a set of official tools that help us with the implementation and automation of tests. In addition, little by little frameworks, platforms, and new technologies are emerging and they are helping us to automate the DevOps process of a voice-based application. 

The voice has reached our homes and devices and that is why we must take ourselves very seriously in terms of software when we create new apps for this kind of platform. Ensuring high-quality thanks to DevOps processes that meet everything a voice-based application requires.

Up next for Xavier Portilla Edo:
I am working on bringing the power of voice on using Alexa and its Skills around the world with tech talks, workshops, and useful posts. Trying to consolidate myself as a valuable professional on this platform.

Our Biggest DevOps Challenge to Date and How We Overcame It

By Jeffrey Fredrick (@Jtf), Managing Director of TIM, an Acuris company. Jeffrey is the co-author, with Douglas Squirrel, of the book Agile Conversations, which describes one route to encouraging and producing a productive conflict of the kind described here.

Our biggest DevOps challenge was the product of earlier success. Over several years, we had developed a new approach to our infrastructure and created tooling that empowered the development teams to quickly get new applications up and running. As part of this work we made it easy to set up all the components that used to be hard: databases, backups, multiple environments, redundancy—it was all there. The good news? Make something easy and people will use it, so it was easy to get developers on board using the new tools. The problem? Making it easier to have multiple instances and multiple databases in multiple environments have resulted in an explosion of virtual machines. These hundreds of running instances have become a major maintenance challenge and added a daunting level of complexity when thinking on the systems level.

The key to our overcoming this challenge has been—and will be because we aren’t done yet—our culture of open communication and collaboration. We explicitly encourage people to challenge our beliefs, and this allowed one of our engineers to raise the proliferation of VMs as a problem that we needed to solve. We were able to have an open discussion among the technologists and then make the case to the business about the need to invest in a solution. As a department, we were able to talk about the many different factors we should consider in a solution, and then invest over time.

We wouldn’t consider this problem solved, however, we continue to openly discuss the current status, how it affects our operations, and make additional investments in simplifying and reducing our VM estate. Key to identifying and addressing this problem was our ability to have productive discussions involving multiple perspectives—productive conflict between ideas.

Up next for Jeffrey Fredrick:
They will be speaking at the DevOps Enterprise Summit London Virtual in June. Learn more.

Siloed Organizations and Siloed Thinking

By James Head (@jimmyrhead), CEO, and Founder at Rebellion Consulting

One of my biggest DevOps challenges to date has been to break down siloed functional/ hierarchical (vertical aligned) thinking to value stream, autonomous (horizontal) thinking.

Many organizations think that agile or DevOps belong to the tech teams or perhaps a tech team plus a product owner or business analyst they fail to grasp how to really unlock the benefits of DevOps the makeup of each team needs to extend its reach much deeper into the organization from the creation of ideas upstream all the way to the delivery of valuable products and services to customers downstream.

There are a number of ways to tackle this problem, the success of which very much depends on the wider dynamics of an organization. I've mentioned 2 of the methods that I've used in the past;

1. The Exemplar:
   In the past where an organization has been willing to give sufficient autonomy to create a new team for a new initiative its been easier to start as you mean to go on and build the team in a manner that optimizes for the flow of work from idea all the way through to delivery to the end customer, I've done this successfully when senior leadership has given autonomy for the team to make decisions about how they'd like to work and supported teams being made up from many domains or expertise.

Works best: In the context of Greenfield teams with autonomy.

Pro's: Easier to get started and less hurdles to overcome, shows results quicker.

Cons: Still sits within wider organizational constraints, can be vulnerable as to cost cutting measures if still in the early stages, lots of hiring/ building the team.

2. Expand the sphere of the team.
   I've worked with numerous high performing technical teams in the past who have been inhibited by the inability to pull inbound work and user/customer needs from the organization, typically because of silo's and handoffs, the aim here is to attempt to reduce handoffs by widening the sphere of the team to blur the departmental lines, by adding a product owner, a UX specialist and asking developers to pick up more testing responsibility, you can gradually decrease the silo’s and handoffs once you have achieved this repeatedly with more areas.

Works best: Mature organizations willing to experiment.

Pros: Change that takes longer and is more gradual tends to be "stickier" in the long run, can retrain and refocus internal expertise.

Cons: Sits within wider organization constraints, can take longer to see impacts.

The ultimate ambition for any team is to be able to take an idea and deliver it to an end customer with no outside interventions, approvals or sign-offs, we want to be able to get real value into the hands of customers as soon as possible and then seek feedback for further incremental cycles. But please remember this is ambition so it pays to be pragmatic, take your time and do what works for your unique context, despite what anyone will tell you there is no magic 5 step process that can be implemented. Like anything else, you need to experiment, incrementally improve and keep learning along the way.

Up next for James Head:
They will be speaking at the DevOps Enterprise Summit London Virtual in June. Learn more.  

Convincing Developers They Should Be On-Call

By Craig Cook (@CraigCookIT), DevOps Coach, IBM

Nobody likes being woken up in the middle of the night to fix a broken IT system. This is normally an operations duty. Developers build a service and someone else runs it. 

We did have some developers leave our teams after this was introduced. We also had some developers attracted to our teams because we became known for great DevOps practices. Being "on-call" was included in the setting expectations phase of the interview process for future developers. 

Once they started the experiment a lot of developers discovered the joy of solving problems in production and designing solutions to minimize call-outs. Having control and access over your production systems is empowering. You quickly get feedback from the impact of your work. Overall service quality jumped due to the sense of ownership and pride.

Up next for Craig Cook:
They will be speaking at the DevOps Enterprise Summit London Virtual in June. Learn more. 

"Biggest DevOps Challenge" and Solution

By Ann Marie Fred (@DukeAMO), DevOps and Security Lead, IBM

We're struggling with a combination of thin architectural documentation and cutesy repo and microservice names. 4 years and 150 microservices later, and nobody knows what many of them do.  We didn't know who owned each of the 1,150 repos in our Github org, and that's important for security purposes.

How Did We Solve This?

●      Part of it was a cultural change; over time people realized why meaningful names are important, so the newer projects tend to have more meaningful names. 

●      A second thing we did was migrate all of our services to a new IBM Cloud account.  After everyone moved their services over, there were a few left that we couldn't identify.  We asked around for a week or so and then stopped those services. Nobody noticed, so we deleted them.

●      Finally, to find repo owners, we put together an initiative whereby every squad had to register their repos with our DevOps Metrics dashboard.  That gave us a mapping from repos to squads.  We did detective work on the unclaimed repos to either find their owners or archive them.

Up next for Ann Marie Fred:
They will be speaking at the DevOps Enterprise Summit London Virtual in June. Learn more.

Doing the Right Thing

By Kolton Andrus (@KoltonAndrus) -- CEO and Co-Founder of Gremlin

I've been in the DevOps world for over a decade. At times as a practitioner, other times as a manager, and today as the CEO of a DevOps / SRE tool. From my experience, the biggest challenge when it comes to DevOps is simply getting people to do the right thing. Especially if you are migrating to microservices and modern architectures, and you're new to best DevOps practices and workflows, it can be difficult to prioritize work and to enable employees to take ownership.

Up next for Kolton Andrus:
They will be speaking at the DevOps Enterprise Summit London Virtual in June. Learn more.

Overall Summary

Xavi: It is clear that the world of voice is somewhat new and we have to work little by little to create DevOps processes that help us to create this type of applications, generating optimal software quality.

What DevOps Challenges Have You Experienced? 

Let us know in the comments! We want to hear your DevOps stories.