Exclusive Interview: Mary Poppendieck on Lean Transformations

DZone recently caught up with Mary Poppendieck, agile thought leader and author of the award-winning book "Lean Software Developmet: An Agile Toolkit," (2003).  In this exclusive interview, Mary talks about the state of Lean adoption in the software industry today and some of the common challenges organizations may face when transitioning to Lean. She also discusses how developers can get more actively involved in improving customer outcomes, the role of the product owner, and how the notion of 'framing' can be used to improve software development processes and overall product quality. 

DZone:  Mary, can you tell us a little bit about yourself and some of your recent work?

Mary:  I started out life as a programmer. Actually I was a programmer before I went back to graduate school and got a master's degree in math. I did programming for quite a few years until I became an IT manager, which was subsequently followed by a transition to product development manager.

While I was an IT manager for a manufacturing plant, I was in charge of the transition to what would be called Lean today – at that time however, we called it “Just-in-Time.” I did this at 3M, which I think is a company that knows more about how to do product development in commercialized products than almost any company around. Eventually I took an early retirement leave. At that point I was looking around for something to do so I ended up managing some software projects. I was appalled to run into this idea called the "Waterfall Method."

I'd been a programmer and I'd been an IT manager but I'd never actually heard of the concept of Waterfall. Most of the software that I wrote was in process control systems (where I was controlling hardware), not in transaction processing systems. I was surprised at all the hand‑offs that went from a person who had the problem in a piece of equipment that needed to be controlled ‑ or whatever substitute there was in transactions ‑ to the person who actually wrote the code.

I got to looking at some of these things in the late '90's. I was wondering, "Well, how can this possibly work?" I don't understand how somebody can write code simply based on a spec somebody’s given to them. I discovered all the hand‑offs didn't work. The knowledge didn't get transferred.

I've always been of the opinion that if you have to write down in great detail what needs to be done, you may as well be writing the code. Anything that was accurate enough to give directions to somebody that was writing code had to be accurate enough to actually tell a computer what to do.

DZone:  In the ideal world, should all software development ultimately fall under the purview of some kind of "Agile" methodology? There has been some recent discussion on DZone about projects where "Waterfall" is more appropriate and projects where "Agile" is more appropriate. Does it really depend on the type of project?

Mary:  I'm questioning the concept of separating design from implementation. There are cases where it might work when really most of the design is done; however, I don't see how, when you separate the concept of design from implementation, that the person who's doing anything complex or needing any kind of intelligence has enough information to make the right decisions. It's not a matter of "Waterfall" or “Agile,” it's a matter that making sure the people who are doing very intelligent and complex work have the information they need to make the right day‑to‑day decisions, hour‑upon‑hour. What surprises me the most is the idea that that information can be handed through paper documents.

DZone:  This would still appear to be the norm in many organizations today – passing requirements, instructions, and even knowledge, using paper documents.

Mary:  If you look into some of the Asian cultures, the idea that knowledge is something that can be written down doesn't exist. There you see much more the idea that knowledge is much more amorphous. There is this thing called "tacit knowledge" based on the idea that knowledge takes time and skill to develop. Learning is not just about writing things down. It's about having ideas in your head. Also, understanding how to use all of your experience in dealing with problems is a well‑understood concept in other cultures. But somehow or other, the West came to espouse the idea that knowledge is just stuff you write down. I don't think that's true.

DZone:  You’ve published some very seminal work on applying Lean principles to software development. 10 years after the ‘birth of Agile,’ if you will, how would you describe the state of Lean adoption in the software industry today?

Mary:   10 years ago Lean was actually a 1990's concept. It was out of date. It was used in the early and mid '90s in manufacturing. The reason that I wrote the book "Lean Software Development" was to take those long‑standing operations principles and think about how to apply them to a development environment. That was almost a decade ago.

Since then I have been constantly interacting with the development community, giving talks at conferences, I keep learning new stuff and writing new books.

If you look at the Agile, Kanban, and Lean movements, they all share this idea that working in a sequential manner (as advocated by Waterfall) isn't such a good idea and that development is a learning process that requires you to work through learning loops.

When I started giving classes just about everyone who took the class would draw a value string maps of a very, very long, drawn out sequential process.

Today, many people coming to our class aren’t familiar with that concept. In fact, I was at a conference a couple of weeks ago and ran into a young guy who had been in the software development industry for about three years and had no experience with this ancient concept called Waterfall.

It just didn't even enter his brain.

The Agile, Lean and Kanban movements have had a big impact on the way people think about the software development process, but that doesn't mean they’ve solved all the problems of developing complex systems, by any means. People are still struggling with a large amount of complexity in their systems and agile rules are a little too simplistic for solving some of these problems.

There's been a big change in thinking about how one goes about software development, and there has been as big a change in our software development environments. As a result, software and systems have become smaller and easier to segment. Testing processes have become much more robust and much more capable of being automated.

DZone:  How does Lean relate to other Agile approaches like Scrum and Kanban?

Mary:  Well, Lean is not a software thing -- it comes from the manufacturing and operations world and it's been around for a long time. The concepts behind Lean are used in many environments from hospitals to retail stores. In software development specifically, I think of Lean as a set of principles, a way of thinking about things. It’s not so much a movement, but a way of conceptualizing how you approach problems.

Agile tends to be similar, but relatively limited to software development. Kanban, generally speaking, is the name of a Lean tool in manufacturing – it refers to a scheduling technique as opposed to a whole way of thinking about things.

So they're different but they tackle the same problem.

Lean tries to look at stuff beyond software, at the whole system.  It’s about thinking from a systems approach. When you use it in software you have a certain view about how you think about things. Scrum and Kanban are specific agile practices and are ways of managing workflow.

DZone:  In the projects you've worked on, Mary, what have been some of the most common challenges associated with Lean transformations?

Mary:  I think of Lean as a management approach. If you start thinking about Lean as a set of tools or think about it as something that's for development only, it doesn't work so well. I think some of the really big challenges in Lean and Agile in general have been the existing governance systems. If you take a look at governance systems, especially in IT departments or very large software product companies, they get to be quite cumbersome and they are very hard to change, even if they're based on different thought processes. People often struggle when trying to move to an Agile environment because they are still working in a governance environment designed for other processes. 

Another area of struggle is with very large legacy systems. The reason why they're legacy systems is almost always because they're big and complex and hard to change. Great, big monolithic, complex, hard to change code bases don't like to be changed. Agile and Lean like to change things. There's a dissonance there and it's really hard to overcome that dissonance.

Another area of challenge is in embedded software. Embedded software counts for over half the software written in the world. It's the area of most life‑critical software.

It's an extremely complex and difficult thing to do because you're interacting with hard hardware, not just software. The concept of iterations being managed by the software environment doesn't actually play so well on a hardware development environment and, typically, the software developed in an embedded environment has to be a servant of the hardware and hardware development cycle.

So thinking processes help embedded software people, but iterations may not. The way you schedule and manage the workflow in an embedded software environment may need to be rethought, so and that's another interesting area.

DZone:  There is often a disconnect between developers and customers.  How can developers get more actively and directly involved with improving customer outcomes?  Should this responsibility rest entirely on the shoulder of the product owner?

Mary:  Well, we have to get over the idea that Scrum has unfortunately brought into our industry: that there's a person called the product owner who makes all the priority decisions about what a development team should do. That's not always the way a product owner role is implemented, but when it's implemented that way it's wrong. A development team needs to be able to be successful. Success has nothing to do with technical success, it has to do with whether or not what they're coding creates success in the business that they're coding it for and the customers that they're delivering it to.

As a developer, the most important thing is to make sure that the team takes responsibility for it's own overall success and doesn't rely on somebody coming in and being the person telling them what to do or how to set priorities.

That's not always possible for a relatively junior developer, but as one becomes more senior he/she can, in fact, wield more influence over what it is that the team is expected to do and how they're expected to understand and interact with the people for whom they're writing the code.

I really think that the most important thing that the software development team has to realize is that it's not about the software. It's about solving customer problems. If the customer could solve their problem without software they would be delighted. Software is not the point. Working software is an interesting thing, but it has to work, that is, it has to solve a problem. The development team has to figure out what that problem is and think about ways of solving it and not only using software. 

The senior members of a development team have to understand that they shouldn’t just take somebody else's word for what it is that's supposed to be done. They have to work with partners, with the people who have the problem, so that they can get to a good enough understanding of the problem. This will lead to better decision making on a day-to-day basis.

Our goal is not to write beautiful code. Our goal is to think about the problems the customers, the people who are going to use the code, have and figure out how to do an elegant job of solving those problems.

DZone:  If my organization is mandating a Lean approach to software development, how does my development environment need to change or evolve to accommodate this?

Mary:  That’s just too hard of a question for me to answer in a generic way because there are so many different development environments and so many different extenuating circumstances. If development environments get to be so complex they become an end in themselves, we have to get past that. And we have to understand that it's not about the software. It's about understanding the real problem to be solved and creating an organizational structure and a communication structure that’s conducive to this.

One of the biggest problems I see along those lines is when products get big. When products get big, relatively monolithic, it becomes much more difficult for the people working on the product to get in touch with the customers because there are just too many people involved.

There is a lot to be said for thinking about how Conway's Law is applied. Conway's Law says that the structure of a system will take on the communications structure of an organization. An organizational structure which is divided into smaller groups, working in individual areas will most likely create a divisible architecture – this is probably one of the things we have to think really hard about when we’re looking to create really great software.

I lived a long time before the Internet came along, and I’ve seen the development of many, many monolithic systems. What happened with the Internet is we now have a system which is very divisible – and it spans the world. If you think about the Internet as a single system, it is absolutely the most massive thing around. But nobody cares when I change my personal website because it's not going to impact them, it's not going to mess up their bank account.

So here we have a really good example of an extremely divisible architecture and system that has been around for a long time. And when people mimic that concept, they also mimic it in their organizational structure. At the development level, they are able to better engage in the ultimate purpose of what they're doing because the system has been split into much smaller sections, where smaller teams can take responsibility.

DZone:  What are some of the biggest sources of waste in a typical software development project?

Mary:  The biggest source of waste is building the wrong thing or building things people don't need.  The second biggest source of waste, I think, are handoffs, because a lot of information doesn't get transferred, which leads to the first waste, building the wrong thing or building things that aren't needed. Perhaps the third source of waste is to be extremely casual about whether or not the code works -- "Well, we'll figure it out later,"  -- as opposed to always writing known good code, and being able to test the code very incrementally as it's built. This means both testing its intent and testing how it works in larger systems.

I think those things are the things that create the most waste.

DZone:  What is ‘Framing’ and how does it apply to software development?

Mary:  Framing is what everybody does when they look at the world. My husband's a photographer and every time he shoots a picture he frames it in such a way that everybody who sees the picture sees what he sees and nothing else.  When you look at the world, you look through your eyes and you see what you have in your frame. Stuff outside your frame, you're blind to. And some people can get wider peripheral vision, but still they're blind to seeing things in a different way because the stuff that they're looking at is not in their peripheral vision.

There are different ways of framing the software development process. What we offer in our books and in our lectures is our take on what we think is a good way to frame the software development process. It's not the only way; other people will have different frames. In fact, no frame is good or bad by itself. Frames are just the way people look at things. They just are. But we think that the frames that we have used are ones that have led to more successful business outcomes.

The first frame to consider is to look at the whole system. You never try to optimize individual pieces. And in software development the whole system is not about software -- it's about resolving customer problems. You've got to focus on what that means, and taking a much broader view of the world than your own little area.

The second frame is to really understand, technically, what it takes to create software that doesn't have defects. We've come to expect in software development that something like a third of our development time, in any release cycle, will be spent doing final testing. We've got to get rid of all those bugs we put in.

And that's considered acceptable. Well, in my frame, that's not acceptable. We have to stop accepting the fact that we expect to put defects into code and start figuring out how to build code without defects. Similarly, when the US automakers started discovering that, my goodness, their Japanese competitors weren't putting defects into cars they had to learn how to think differently about the whole concept of building cars without defects.

The third frame is the whole concept of scheduling. Scheduling has been thought of in software development as a project management mechanism whereby you lay out all the tasks, estimate them, roll that up into a schedule, and then execute on that. I think that's wrong. I think a better frame instead is to think of work as a flow and to manage that flow at a given cadence. When you do that, you actually get much more reliability, and predictable delivery.

Lastly, there is the frame that we should establish procedures and follow them; however, my frame says there's no such thing as the best way to do something. No matter what we're doing, there's always a way to get better. And the people who understand the situation the best are the people who are doing the work.

In this frame, it's the people working on the ground that are doing the work who are constantly figuring out how to improve the way that they work with mentoring and coaching from their leaders.

DZone:  What general advice would you give to software development shops looking to transition to Lean software development?

Mary:   I think the most important thing is to see the world through the eyes of somebody who is working to design or write or test code. If you think about that person, somebody that's kind of new, what do they need? They need somebody that makes sure they learn how to do their job well. They need somebody to help them understand what's the next most important thing to do. They need somebody who's going to care about making sure that they get the right kind of job assignments that will help them develop their full potential.

These are the three leadership roles that will help your junior developer do a good job:

1.     Somebody that can guide help a junior developer not only a competent developer, but a confident team leader.

2.     Somebody who understands how to put a team together to create a good product. Think like a band director who puts a group of musicians together and creates great music. Each one of those has their own music teacher, but the band director brings all these musicians together and creates a great product out of it.

3.     Somebody who’s going to watch out for that person's career and make sure they get the kinds of things that are going to develop them to their full potential.

Those three leadership roles are important to think about. How do we provide them so that that person who's doing development gets the right kind of leadership and guidance?

DZone:  Mary, on behalf of the DZone community, thank you for your time today.

Mary:  Thank you so much.