We have been doing Continuous Delivery at Wix for several years now and as part of that, I have been training my team at the methodology of DevOps and changing the company’s culture. Last week I was trying to explain to some colleagues how we work at Wix and about our dev culture. While doing it I realized that we are doing much more than just DevOps. What we actually do is something I call “developer centric culture” or in short “Dev Centric” culture.
What is Dev Centric Culture?
Dev Centric culture is about putting the developer in the middle and create a support circle around him to enable the developer to create the product you need and operate it successfully.
Before I go into details about Dev Centric culture and why is it good, let’s look on the problems we have, what we are trying to solve, and why it is more than just DevOps.
DevOps is about breaking the walls between developers and operations, but after a few years of doing DevOps and continuous delivery we realized that DevOps is just not enough. The wall between developers and operations is just one wall out of many. If you really want to be agile (and we do), then we need to break down ALL the walls.
Let’s take a look at a traditional development pipeline.
- Product that “tells” engineering what they need to do.
- Software architect designs the system and “tells” the developer how to build it.
- Developers write the code (and test it if they do TDD).
- QA checks the developers' product and cover it with more tests.
- Operations deploys the artifact to production.
So what we have here is a series of walls. A wall between the product and engineering, a wall between engineering and QA, and of course a wall between engineering and operations. Sometimes there is even a wall between the architecture team and developers.
When a product manager hands over a spec to developers, the developers usually don’t really read the whole spec, they misinterpret the product manager’s intention, they lack the product manager’s visions, and they are also disconnected from the client’s needs.
When you have a team of architects doing all the design and telling developers what to do, you again have a wall because developers might not understand the system’s architecture and will not build what the architect intended.
When developers hand over their product to QA you create another wall because now QA needs to guess how the system should work, and there is also a shift in responsibility where developers don’t feel ownership and responsibility to the quality of their product.
Then comes the last phase, where Operations gets a hold of the product and they are responsible for deploying and monitoring a product they know nothing about.
So when something bad happens there is no one who is actually responsible to the product and you start with a blame game.
If the product is not what the customer expected, then the product manager blames the developers because they did not develop the right thing. On the other hand, developers blame the product managers for not defining the product clearly. If there is a bug (God forbid) on production then developers blame QA for not finding the bug, and QA blames the developers for writing a buggy software in the first place. If there are stability issues in production then everyone starts pointing fingers at operations, who do not maintain the product properly; and on the other hand operations blames the developers for writing unstable software.
So as you can see there are many walls in the process that we need to break in order to have better quality.
So let’s see how we can break these walls, and how “Dev Centric” culture helps us with that.
Dev Centric culture means that you put the developer in the middle and create a support circle around him or her to enable the developer to create the product you need and operate it successfully.
In a “Dev Centric” culture the developer is responsible throughout the product lifecycle, from product definition to maintaining it in production.
Now if you think about a very small startup the developer is the key, and does everything from talking to customers, defining the product, designing the system, writing the code, and deploying and maintaining it in production. This is also the phase where the startup is the most agile and progresses very quickly. Now if the startup succeeds and grows, more people join the company and slowly build walls around the different departments that eventually hurt productivity.
Now let’s see in more detail what walls we have that need to be broken down, and how “Dev Centric” culture can help break them down and improve the quality and agility of the whole organization.
Wall #1: Product || Engineering
When getting a requirement to design a new feature or system, the product requirement is defined by the product manager. The product manager sits with the clients, understands their needs, and translates the needs into system requirements. The problem with that is that the product manager does not write the actual code and design the system. The ones who are doing that are the developers. If developers get a design document, many times they don’t understand the actual customer need, they misinterpret the product manager's thoughts (and between us, they don’t usually read the whole product requirement document). This causes a waste of time because the developers might develop something completely wrong that doesn't meet the customer's needs. If the product manager catches it in time, the developers can fix the application before it gets to the customer, but it just wasted their time because they could have built it well in the first place.
In Dev Centric:
Instead of product manager tell the developer what to do, the developer sit down with the product manager and TOGETHER they define the product vision. This way, the developer gets to understand the product and the customer’s needs. By understanding the product, the developer will create a much better system that fits the client’s requirements. If we add to that Continuous Delivery and “lean startup” concepts, the developer is a very good resource to help the product managers define the MVP (Minimal Viable Product) and make sure that no feature is being developed that is not needed.
Wall #2: Architect || Developers
In many organizations, once the product manager is finished with the product definition, the “ball” is in the architect’s court. The architect designs the system and hands the design to the developers to develop the application. In this scenario, the developers don’t completely understand the architecture and may not produce what the architect intended. Another side effect is that we treat the developer as a coder and not as a software engineer, thus limiting their ability to cope with the complexity of designing a good product. This also creates a big workload on the architect who now has to supervise the engineering work and make sure they follow the design.
In Dev Centric
Instead of handing the designs to the developers, we need to invest in the developers' growth and teach them how to design a good system. Architecture should be implemented by the developers themselves while the architect should become a consultant and a supervisor. There is a challenge here for the architects because in many cases there is not one correct architecture, and the developer might choose a different architecture than the architect would have chosen. The architect should be open to accept a different approach as long as it is a good one, even if it is different from what he thinks. Architects should mentor the developers in how to design a system, teach them how to approach a problem, how to analyze it and guide the developers to think about the architecture concerns that developers are not aware of or missed.
By doing this you invest in your team and grow their knowledge, quality and abilities, thus creating a better product and better developers.
Do not be afraid let the developers make mistakes. As long as they learn from the mistakes you will end up with better engineers. If architects feed the developers with the answers all the time, the developers will not learn to cope with complex problems on their own and will come to the architect for every problem, thus creating a much bigger workload on the architects.
Wall #3: Developers || QA
When developers finish their work and hand it over to QA they are being put in a state of mind that they hand over the responsibility of ensuring the quality of their work to someone else. Instead of owning the product that they have just written, it is up to somebody else to find their bugs. This case causes developers to not invest too much in the quality of their code and not feel responsible for their own work quality. Now when a bug is found in production, the developers believe that it is QA's fault for not finding the bugs.
In Dev Centric:
The solution for this problem is really simple. You need to make your developers write tests and be responsible for their own work. The only way to do that is to start doing Test Driven Development. Developers who work in TDD write better code and their designs are much better because it has to be testable, with the right abstractions and separation of concerns.
Now for developers who are not used to writing tests this can be a challenge, but from my experience, once they do that long enough, the developers understand the benefit of TDD and never want to go back to not writing tests.
Developers should write both unit tests and integration tests. For back-end development this is usually enough, no QA is necessary unless in very rare cases where changes are very deep and might affect other systems. At Wix the back-end group works without any QA.
Now you might think that if developers do their own testing, what do we need QA for. Well you still need QA, but QA should change their role. Instead of verifying the developer’s work quality they should complement the developer’s tests by writing automated system tests (end-to end) and in some cases do manual QA for UI components during development until an automated test is created for them.
Wall #4: Developer || Operations
In order to deploy the application, the developers or QA hand over the artifact to operations and the operations people install or deploy the application. The problem is that the operations people know nothing about the application, what it does, and how it should behave. Now we expect them to install it and also maintain applications that they are not familiar with.
In Dev Centric
The developer should be responsible for their own applications in production. The developer should install and maintain the applications with the support of the DevOps engineers. DevOps engineers should create the tools to help the developer install and maintain their applications on production. While developers should have ssh access to production, ops should create the tools so that developers will not have to ssh to production.
Make the production environment visible in a developer friendly way. Via a monitoring system and easy access to logs (log aggregation), developers should see how their system behaves in production at all times. This creates a better quality product because now the developers are also responsible to their work in production. Every time there is a problem in production, the developer should be alerted and handle it in production with the help of operation engineers.
Wall #5 Developer || BI
We see BI as a business concern and generate reports to marketing and management, while the developers are not exposed to most of them. This creates a lack of knowledge of what is going on with the system and how it is used by the customers. If the developer does not know how a system is used they will not be able to make improvements on the parts that really matter.
In Dev Centric
Expose the BI reports to the developers. Make the BI tools accessible to developers so they can add technical data and take advantage of BI analytic tools to understand how the system is used. By doing that, the developer can have a sense of priorities and knowledge of how the system is used in real life. Once developers understand that, they can make improvements and optimizations to the most heavily used parts of the system and by doing that, help customer satisfaction.
Maintaining a Dev Centric culture is not easy. You put a lot of responsibility in the hands of the developers. You actually give all responsibility to the developers. But if you think about it, this is exactly what you should do. The developers are your company assembly line, they are the ones who actually create the product. They have a direct impact on the company’s success and bottom line. By understanding that and entrusting the developers with the power to change and take critical decisions it is a win-win situation.
Remember developers CAN DO the work of a product manager, architect, QA and Ops, in fact this is exactly how a small startup succeeds, because developers actually do all of it.
What do you think is it time to go back to the future by breaking all the walls?
Next Chapter: Dev-Centric Trust and Collaboration