The Need To Review The Agile Manifesto
In 2001, when the Agile Manifesto was first published, technology in the workplace looked and felt very different to how it does today. High-speed Internet and data channels were still relatively new and communication channels more limited in variety. The sharing of information was nowhere near as easy as it is today. Nothing lived in the cloud and ‘drop box’ was something you did by accident, if you were clumsy.
There’s no doubt that the Agile Manifesto has been great for software teams, but it has always presented challenges for business analysts and other stakeholders involved in the broader context of application development. The problems addressed by Agile in 2001 still exist today and are, in many cases, magnified by the changing landscape of product delivery, specifically:
· More information than ever is available, maybe too much;
· Context, conversations and decisions go undocumented;
· Communication gaps widen due to geographically-dispersed teams;
· Time to market has dramatically shortened;
· Customer needs continue to go unmet.
Many of us in the software business have lived through the challenges faced by
Waterfall, Agile and hybrid teams alike, both from the technology and business sides. With great respect for the Agile Manifesto, in light of the seismic shift in technologies and the way in which we work in 2014, I believe that it is time to reconsider the Agile Manifesto, to rethink how it applies in the modern product development dynamic.
Motivations for change
So what does it mean to take a modern look at the agile manifesto? Why does it matter? In my view think there are four key reasons for change:
(1) The world has changed
The world has seen many fundamental changes since 2001, not least in terms of technology and how teams work. Working patterns and methodologies have changed. The workplace has truly become global and connected. From online cloud storage to social networks, we’re now permanently embedded through modern technology. Not only is this true for how we communicate with colleagues, it is true for how we connect with customers. As a result of the social communication revolution, hierarchies of communication have been flattened and reshaped power relationship between rulers and masses (or management and workers). The dialogue with customers is instantaneous, and the customer voice is more empowered than ever before, more able to share opinions with us and everyone else about a product or service.
If we take a snapshot of 2001, the world of technology was a very different place. For example, the Ericsson R380 had just been released in 2000, widely considered to be the world’s first ‘smartphone.’ The first camera phone was rolled out in Japan in 2000 by J-Phone, using technology that would not find its way to North America until 2004. Social media was really an unknown term. Even MySpace was not around yet and the closest we had to what we would now call social media was archaic Instant messaging applications such as ICQ.
Music technology was probably the big event of 2001. Napster had just taken the industry by surprise and the first iPod had just come out. Things would never be the same. The phrase ‘DotCom’ was still something of a buzzword, and AOL was at its peak user base. Google had 400 employees worldwide and the whole idea of a creative workplace was just beginning to emerge. Amazon turned its first profit, laying the foundation for major growth in the coming years.
Today things look very different…
Mobile phones are no longer just phones. They are wearable. They are tablets, music players, movie players, video and still cameras, navigation systems, mini computers and health monitoring devices, and much more, dependent on which of the tens of thousands of apps available you use. Social media platforms such as Facebook, Twitter, Pinterest, LinkedIn and Instagram have fundamentally changed how we communicate and share media and information. Content is highly personalized and we are firmly in the age of opt-in and permission-based marketing and communications.
The ‘Internet of Things’ has replaced DotCom and almost everything is connected - refrigerators, thermostats, cars, smart TVs. And, of course, we know what happened to Google.
There are a couple of important points to note from all of this. One is the sheer number of companies that have emerged over the last few years, but the more important fact is that so many of these companies are making products which have software embedded in them as a core operational element.
(2) Software is everywhere
The second point is that, today, software is everywhere. It’s in everything we do. We as consumers interact and buy software more than ever, often without even knowing it.
In late 2013, Jama partnered with Forrester Research on a ‘modern product delivery’ survey that found that 23% of products now contain software in some form. ‘Disruptive’ technology, that is, innovations that create and markets and value networks, and which over time ‘disrupt’ existing established markets, is increasingly pervasive. For example, Amazon changed the book selling – and everything selling industry - with software. Airbnb is disrupting the Hotel industry. A new generation of tech savvy consumers is helping to drive social adoption and they see disruptive technology as a positive thing.
Established industries are also evolving to adapt software to improve performance. The automotive market, for example, used virtually no software back in 2001. Now we have software working all over our vehicles, from sensors and cameras to navigation to infotainment systems to diagnostics. In 2001, cars had a minimal amount of code in them. A new car now has about 100 million lines of code. What’s more, it is expected that more than 150 million connected cars will be on America’s highways and byways by 2020. We have already seen examples of software being used ‘on the fly’ in remote product recalls. In March 2014, automotive manufacturer Tesla addressed a known fire risk in its cars in part by providing a software update to existing vehicles. This helped mitigate the risk without owners needing to visit dealerships or service centers. Cars are also beginning to connect to each other. The U.S. Department of Transportation is working on a system that has the potential to reduce accidents as a result of an intelligent connected car network.
In the financial industry, it’s becoming less and less important for consumers to have face to face interactions at their bank because of mobile banking, deposits using your camera phone, apps that allow you to budget and transfer money. This has banks deeply concerned. They were not prepared to become a software company. Now they have no choice.
(3) Complexity has increased
The third reason for reviewing the Agile manifesto is simply that complexity has increased, both at the product and company level. Building products used to be simpler as hardware products could only handle so many lines of code, and web applications only had to worry about a couple of browsers and monitor sizes, and there were fewer programing frameworks.
In the modern product delivery survey conducted with Forrester, it was found that 55% of companies had over 100 products and 87% of companies had multiple teams working on projects. 70% of companies stated that they release products quarterly or more frequently. 61% of projects had at least four different teams involved, while only 4% of stakeholders were co-located in close proximity. 23% of products now consist of both hardware and software elements.
Today, products are more complex by their very nature, often performing multiple tasks simultaneously, and generally in a smaller form factor. Hardware contains a lot more software; software products need to consider many more devices and situations; and open source has provided the development community with many more libraries, frameworks and languages from which to choose. The range of products available is wider and product development has become more complex. A greater number of platforms must be supported – in 2001 Firefox and Chrome did not exist yet, now there are many browsers, as well as many different devices and platforms on which to browse. That’s not to say there was no complexity back in 2001, but there have been significant added layers of complexity since then.
Many products now have a much greater ‘ecosystem,’ in that they operate in conjunction with other products to enhance functionality. For example, the success of Apple’s iPad was around its ecosystem, that is, the applications that it enables and which enable it with greater functionality. Nest’s vision was not just around home thermostat but more around the broader ecosystem of home control which, bigger picture, was what Google was interested in when they acquired Nest.
When we look at the complexity of companies, technology has progressed so much that we now have the ability to communicate across distributed teams in many ways that were unavailable in 2001, enabling geographically dispersed teams to work together more efficiently, but which can also amplify the organizational challenges of product delivery. There is less a need for everyone to be in the same room, and many organizations have product teams both here in the United States and overseas.
(4) Projects still fail
And lastly, Projects still fail. Why is that? Why can’t we get to a point of 100% success?
One study, published in 2012 by Dr.Dobbs reported that Agile had a 72% success rate, compared to a 64% success using traditional methodologies. While better, an 8% improvement is hardly a revolution. In today’s competitive business environment, we need to do better in terms of success rate. A further study, conducted by McKinsey, reported that half of IT projects with budgets of over $15 million dollars run 45% over budget and deliver 56% less functionality than predicted.
Put simply, Agile is not a silver bullet. Projects still fail at roughly the same rate today as in 2001. It seems little has changed or evolved in this respect. Looking beyond the numbers, there are several reasons why more than one in four projects fail:
- Teams are out of sync. This is not just about development teams. It’s about cross-departmental teams, from business to engineering, executive to employee. It’s about the company’s entire ecosystem. An engineering team being agile only makes them agile but does not make the company agile. The process or methodology selected should take into consideration the entire organization.
- Secondly, as we’ve alluded to - there really has not been any silver bullets. This is an historic reason, but nonetheless still relevant. The problem is that we continue to seek out the silver bullets - the ‘killer app’ - and fail to recognize the nuances of human behavior and complexity.
- Decisions, decisions, decisions - Decisions have an impact, and there are hundreds, if not thousands, of decisions that take place during a product lifecycle. Robert Goatham, in his paper ‘The Story Behind The High Failure Rates In The IT Industry’ outlines the number of individuals involved compounded with the complexity of the project as primary reasons IT projects fail more frequently than others.
- Lastly, people matter. This is a cliché, but it is 100% true. You could have the best process in the world and if the people are not invested, or don’t care or don’t get along, you are at risk. It’s not to say that people are the only thing that matter but it’s a very important factor in a project’s creativity, innovation and success.
In addition to the reasons mentioned above, the whole idea of failure has shifted. The Agile Manifesto promoted the idea of working software over big bang releases, but working software can have different meanings in organizations. Defining product success is vital. It’s important to establish what it means in yours. Evaluate the correct amount of context you need to provide your stakeholders to ensure you can maintain a working software process. Working software is no longer about just getting half-baked increments to the customer with a means for review. Moreover, it’s about providing something viable; something tangible and which adds true value.
Even failure can provide useful data. If analyzed correctly, the data collected from failure can be valuable and actionable feedback. Product management is not only about defining what needs to be built but also about designing customer and stakeholder expectations. By defining outcomes and managing expectations with the rest of the organization, you’ll be better able to utilize the data obtained from product failures.
Success is theoretically about getting it right. Yes, agile and the idea of working software is about putting it out there for feedback, which means you are not always going to be right. What’s key is to be prepared to react to the feedback so you can act quickly and continually. This builds loyalty and admiration in a world where customers are highly engaged and demand interaction.
2014: Is the Manifesto Agile or is Agile the Manifesto?
Taking all of the arguments and mitigating factors for reviewing Agile into account, what does today’s Agile look like? Do the values of the Manifesto still apply? When did Agile become more about process and less about the mindset? How can we merge the intention of the Manifesto with the new way we work? How can we evolve Agile concepts to tackle the challenges of today in a new way? To what degree have we relied on others to define what works for us? How has the world changed, such that the concepts of agile or the methodology should adapt? One idea to consider is the notion that, in 2001, the Agile Manifesto, in attempting to describe one element OVER another (albeit with a disclaimer at the bottom) may have caused some of the confusion. In 2014, we should focus on elements COMBINED WITH or in PARTNERSHIP WITH each other.
Tools, Processes, Individuals and Interactions
In 2001, we worked more independently and our workplace interactions were limited by the technology of the times. Waterfall at its core was designed to limit interactions and focus on set, defined goals. Agile’s goal was to encourage teamwork and collaboration, and because technology and communication were still relatively limited, and working closely together was facilitated by teams sitting physically close together. This did not scale well across many organizations and roles such as business analysts and project managers were often left out of this process.
Process and tools meant something very different in 2001. Tools were few and process was very heavily manual with lots of steps and detail. Process focused more on quality than it did on getting the product right. So the product might not be fully effective, but boy did it look nice. The tools didn’t really drive process in the way they do today. Rational was driving Rational Unified Process, but tools were immature in comparison to those that exist to support the process today. This is a perfect example of the Manifesto favoring process over tools.
In 2014, the complexity of products and teams has increased significantly as have the channels we use to communicate. Our understanding and acceptance of interacting through a medium has dramatically changed and continues to change. There is greater collaboration right along the product ‘food chain’ and tools have matured greatly and are integrated and valued more than they were in 2001. In fact, today’s tools and processes are tightly coupled, or combined with, with individuals and interactions. It’s a partnership.
Some argue that the reason agile teams fail is that they did not follow the process strictly enough. This logic goes against the Manifesto. Process is an important consideration. It’s impossible to not have some kind of process. Tools are important and today it’s the combination of tools and process that is most important
Working Software and Documentation
In 2001 the notion was that documentation should be replaced by working software. Of course, back then ‘software’ was a simpler concept. Certainly some was very complex, but overall, software products have grown greatly in complexity. The mindset at the time often was to document everything upfront, then go build, the result was that teams built the wrong thing.
Alistair Cockburn, signer of the Manifesto, has spoken about the word “comprehensive” and the decision to use it. According to Cockburn, this software term was highly debated. The creators didn’t want people to think that documentation in and of itself was unnecessary because they did believe it was important. The intent was to call out exhaustive documentation as overkill.
Today we have more complex software. We also have realities of Minimum Viable Product (MVP) and continuous delivery. The idea of working software is much more of a reality. This does not change the importance of documentation. What does change is the idea of the word ‘document.’ A white board, sticky notes, wiki, or collaboration software, these are all documentation. This is a critical and necessary aspect of the process. The ability to respond to change, to interact, in fact everything the Agile Manifesto believes in relates to communication and collaboration around something - the ideas, stories, epics, and decisions written and made everyday.
Customer Collaboration and Contract Negotiation
At the time of the writing of the Manifesto in 2001, the expectation in the software world was that customers and developers defined budgets and timelines of what was expected to be delivered and when. The developer was then obligated to deliver to specifications and on time. This arrangement left little room for change to overcome the challenges or leverage the opportunities that inevitably arise during the development process. Contracts may have kept some projects on track but they also stifled the flexibility to add features or improvements that may have served the customer well. The Manifesto’s solution was to have the customer onsite in lieu of making contracts. This was a noble idea, but not practical for all organizations or industries.
The reality is that in some companies and industries, contracts are still necessary. Sometimes contracts are part of the vendor selection process or related to regulatory requirements. Additionally, having the customer onsite often results in the unintended creation of ‘proxy’ customers, who are supposed to convey the concerns and interest of the customer. Instead, proxy customers created a point of failure. Relying on one individual to understand and comprehend what customers need and want can result in communication breakdown.
In many organizations, the role of business analyst helps to bridge the business and the technical sides, to translate the customer’s problem. They can help define what the business problem actually is which might not be in the same words a customer would use to explain needs.
Today when we say ‘collaboration’ it means something different than it used to. Modern collaboration tools make it easy to convene, discuss and agree to decisions with individuals around the globe. Collaborative processes are now a reality for dispersed teams. We can be in constant communication with our customers.
While collaboration has not eliminated the need for contracts across industries, it has changed our ability to combine collaboration with the contract. The contract keeps teams in alignment with the original vision and core business value of what is being built. If the vision changes continued communication with the customer means the contract is a living agreement that can adapt to change.
Responding to Change and Following a Plan
In 2001 the perception, and often the reality was that a plan represented the final work. It was treated like a gate into the black box of engineering. You had to have a plan or you couldn’t solve the customer problem. The engineering team then went into the ‘Black Box’ with the plan and built the product based on their interpretation of the plan. Of course when the product failed, many blamed it on a faulty plan. Others blamed engineering for misinterpreting the plan.
A lot of effort went into defining requirements upfront so they were crisp and clear to the developer team. Entire books have been written about how to write requirements in an effort to reduce confusion around ‘the plan.’ The Agile Manifesto tried to fix this by eliminating documentation and moving away from ‘the plan.’ Teams also began to interpret the idea of responding to change as meaning ‘just do what the customer wants.’ How you respond to change is key to determining whether your outcome is success or failure. Understanding how to get to the core of the change can be an indicator for success.
Change is inevitable. Regardless, there must be a plan. It’s important to refer back to the plan often to avoid drifting away from the core business value your product needs to deliver. Any drift needs to be within acceptable bounds for the project and is necessary, if it isn’t then the plan needs to evolve.
Ultimately change and vision go hand in hand. We must work together to constantly evaluate and understand what we are building and why. One can even argue that it’s not change we are responding to. Change results from the many decisions made throughout the process. People can’t make good decisions in a vacuum, they must have a clear understanding of the business value they are trying to deliver and a crisp grasp of customer needs.
Looking to the Future
I believe that the future is something that we at Jama refer to as ‘Modern Product Delivery’ or MPD. We now better understand the ways in which the world has changed and how to redefine how we discuss process. Ultimately it’s about a process that involves the entire organization and takes into consideration collaboration, people and organizational structures within the organization. It’s not just about engineering anymore… it’s about everyone.
So, what are some specific changes we can expect? Methodologies are less about process and more about organizational structure.
- Data and Science Will Drive Decisions – We’re already seeing this We’ve heard about big data and connected products are helping companies better track and understand how their products are used and why. For example, Tesla is gathering a lot of information about their current fleet in order to continually make software improvements and evolve future models. Contract data can provide incredible value for future versions of products.
- Culture will continue to be a focus - It’s one of those things that successful companies take seriously. It’s not about video games and smoothies; it’s about working together in an empowered, innovative way.
- Individual behavior will be a driver of the future of work - Digital fluency, or the ability to harness modern technology and tools to achieve better results, will become increasingly prevalent in the workplace. Younger generations are able to interact in many different ways with ease and don’t want to rely on decades old technology to work.
- Solutions and tools will take a larger role - Our world is surrounded by software and people almost demand it now. White boards are good for meeting conversations but they are not the place of record for complex projects. You need tools and software to accomplish that.
Taking into account everything that has been discussed above, I’d like to suggest some recommendations to ensure that product development is optimized to give organizations the best chance of success and customer satisfaction:
1. Pause and Rethink
- Decouple the manifesto from Agile – take a moment to consider the manifesto through the lens of your own surroundings and look your own processes.
- Continue to shift the understanding of failure – Look at how you’re releasing information and reacting to the release of that information. If you’re providing some kind of working software, it’s important to understand how you’re reacting to that. What are you doing as an organization to react to that information?
- Evaluate your current communication modes – Are you an organization that loves to have meetings? Do you love to talk on IM or email? It may be worth actually recording some data on this and do some research and looking at the results – number of emails and reply-alls. You may find that decisions are being made in those vacuums and people are being left out because they weren’t even aware of a conversation that was happening.
- Track metrics – If you do not have metrics, then it’s an emotional game deciding which direction to go in. One risk you run with metrics is misinterpreting the data so be sure to debate then negotiate what the metrics mean.
- ·Think of Agile from an organization perspective - The more we can spread the idea the more it will mature.
- Find ways to constantly provide visibility – Into what you’re working on – even if it’s in its early draft stages. Sharing early and often is a good thing.
- Find ways to allow communication and opinions to flow freely - There can be a fear that the noise level will hinder progress but I have not found that to be the case. I think there are ways to allow information to flow, and if it is found to be overwhelming, you can always work backward and filter it down to the relevancy of it. Having the wealth of information is a wonderful starting point. Your customers could be interacting with you through hundreds of mediums. It’s not valid to say you wish they would not interact with you but more valid to distill the information down to its relevancy.
- Identify where decisions are made and captured - Decisions are happening everywhere and they are what cause your product or project to drift – going away from the initial path to the objective for a number of reasons. This will continue to become increasingly important.
- Constantly ask if people understand what they are doing and why - There is an opportunity from a company perspective to empower and motivate by enabling people at every level to understand what they are doing and why. It drives creativity, innovation and success.
- Open up the dialog about culture - If you think this is not your role than think again. If you’re involved in the development and delivery of products you have a perspective and a voice.
- Embrace process - Focus on making the process more efficient and understandable. Process gives us guidelines to collaborate more creatively and effectively.
- Continue to communicate – Content and data is incredibly valuable if it’s made available in the right place. Having a ton of information is a great starting point, the next thing is figuring out how to utilize it.
- Start with too much information then work your way back - Find the balance. In early stages of a product this can be especially true. Having more information will help you eventually develop parameters.
 Source: http://www.washingtonpost.com/local/trafficandcommuting/direct-communication-between-car-computers-may-reduce-accidents-by-up-to-80-percent/2014/02/03/b55e9330-8d1a-11e3-833c-33098f9e5267_story.html
Source: All figures from The State of Modern Product Delivery Report, Forrester for Jama Software, December 2013