Patterns of Effective Delivery (Dan North)

The following are some highlights from Dan North‘s excellent, inspiring, and insightful talk Patterns of Effective Delivery at RootConf 2011. North has a unique take on what agile development is, going beyond the established (and rather rigid) views. I really recommend this talk to learn more about effective teams, about North’s “shocking,” beyond-agile experience, and for great ideas on improving your team.

The talk challenges the dogma of some widely accepted principles of “right” software development such as TDD, naming, and the evilness of copy/paste. However the challenge is in a positive way: it makes us think in which contexts these principles really help (in many) and when it might be more effective to (temporarily) postpone them. The result is a much more balanced view giving you a better understanding of their value. A lot of it is inspired by the theory (and practice) of Real Options.

What are Patterns of Effective Delivery?

• Patterns – Strategies that work in a particular context – and not in another (too often we forget the context and to consider the context where a strategy doesn’t work / is counter-productive).  Beware: a part of the context is the experience of the developer.  For inexperienced devs it might be better to just stick to a process and apply TDD all the time instead of trying to guess when they it is appropriate and when it is not.
• Effective – Optimize for something: Volume of software produced? Time to market? Learning/discovery? Certanity? User experience? Any of these will work.
  
• Delivery – Get stuff that is useful out of the door. Software is not important, the utility it provides is. Know why you write the software.

Some of the patterns take years to master and require significant investment before you start seeing the benefits. You might need to fail a few times before getting them right.

Disclaimer: These are notes that make sense to me. They will likely make only limited or no sense to people that haven’t heard the talk. It would be best to go and listen to it instead.

Selected patterns

Spike and Stabilize (or throw away): traditionally we decide whether we are writing production-grade code (with high rigor such as TDD) or just a throw-away spike before we start coding – i.e. at the moment when we know the least about it. We should not decide this up front but “exercise the option of investing in the quality” later, based on experience. Start as a spike and if the code proves valuable, stabilize it, refactor, test etc. Evolve the code based on experience (good naming, quality).  Defer the commitment to the quality of the code and optimize for learning

An example of spike-and-stabilize regarding test naming: take a test originally named blah – you don’t know what it should do yet but you're experimenting. When the code evolves into something meaningful, name it properly, according to that.

Ginger Cake – Copy and paste code, rip irrelevant things out until only the important things are left, then write tests around it.  You may end up with code that is similar, but not in the ways you expected. If you started with abstracting, it would be the wrong abstraction. The pattern says: “We know and respect DRY but are not slaves to it.”

Short Software Half-Life: 1) We don’t care about the software but the utility it gives us. If writing it gives us better ideas, we can delete it and do the better thing.  2) How would you write the code if 1/2 of it – but you don’t know which half – would be gone in a few weeks?  The answer is, start simple (see Spike & St.), extract commonalities, improve quality, etc. For code that has already been around for a while and has proven itself useful; Some architecture styles lend themselves better to such quick evolution – such as small, focused services, popularly known as micro services (see slides, esp. p.42+).
“Look at the code as it evolves and decide what to invest in.” (The investment includes thinking about the design.) All code is not equal.

Create Urgency – To change a paradigm, the way of thinking, people must be desperate and have no more options along with the knowledge of what to do.  Apply this pattern when learning something new.  Do it on something real, under self-inflicted pressure.  For example, you could commit to do an app with a crazy deadline using some new tech.  This would give you a sense of urgency, with no more options. It forces you to learn only the parts you really need.

Socratic Testing (coaching style) – Don’t tell the team what’s wrong with their code, which is threatening and thus hard to accept. Pair with them on writing a test and to support the test, make “helper” classes etc. that you’d like to see in the production code. If they really are useful, they will spot it and decide to pull them into the production code. Make them the hero. Respond to their questions with another question.

Fits In My Head – we need code that we can understand and reason out (big classes, methods, complex models, etc.). Keep the code simple, optimize for understandability, readability, and obviousness. Build Shared Idioms in the team so that the team members would, given the same context, arrive at the same decisions/design. Something should only differ from the usual way of doing it when there is a good reason for it.  Thus a difference provides a hint, difference is data. For example, putting all communication over ZeroMQ at only one place through shared memory. This indicates there is some (most likely performance) reason for it.  Communication strategies shouldn’t be picked randomly or ad-hoc.

TDD – A pattern that, in a particular context, may make you much more effective.  Most of you reading this know what TDD is.

Bonus: Micro Services

Rough notes from James Lewis’s talk Micro Services: Java, the Unix Way (2013) – especially slides 42+:

• Use web, do not bypass it – REST, JSON; standardized application protocols and message semantics
• Small with a single responsibilities (does one thing, fits into one’s head, small enough to rewrite and throw away rather than maintain)
• Containerless and installed as well-behaved Unix services (executable jar with embedded Jetty + rc.d start scripts and config files)
• Avoid unnecessary coupling; Domains in different bounded contexts should be distinct; It's ok to have duplication with physical separation to enforce it; There will be common code, but it should be library and infrastructure code; Leverage Conway’s Law to support decoupling
• Provisioned automatically: “The way to manage the complexity of many small applications is declarative provisioning” (including instance count, scaling, load balancing)
• Status aware and auto-scaling; In-app status pages; monitoring to autoscaling
• Each service is entirely decoupled from its clients, scalable, testable and deployable individually

Decomposition: This technique goes from product to a set of capabilities (e.g. monitoring, reporting, fulfillment, user) and then to each capability being implemented by a set of small apps/services exposing a uniform interface of Atom Collections. The capabilities form the project by interacting via a uniform interface – HTTP (reverse proxies etc.), HATEOS (link relations drive state changes; its an anti-corruption layer that allows the capability to evolve independently of its clients), and standard media types (usable by many types of clients).

Explicit tips from the talk:

1. Divide and conquer – Start on the outside and model business capabilities.
2. Use Conway’s Law to structure teams (and enforce decoupling).
3. The Last Responsible Moment – Don’t decide everything at the point when you know the least.
4. Be of the web, not behind the web.
5. If something is important, make it an explicit part of your design (reify) – an exampoe would be an instance of services creating users by posting to /user.  They post a user creation request and get response immediately.  The user is then created eventually (reminds me of futures).
6. Favor service choreography over orchestration.
7. Use hypermedia controls to decouple services.

Some tools used: SimpleWeb/Jetty, Abdera for Atom, Smoothie charts (JS charts for streaming data), Coda Hale’s metrics, Graphite. Ops: Fabric with boto, AWS, Puppet, … .

Remember there are NO SILVER BULLETS. This stuff is hard.  Versioning, Integration, Testing, Deployment and eventual consistency are hard for people to wrap their heads around.

Note: Comoyo.com, powered by a number of ex-googlers and other smart people, does the same thing. So does Netflix, I believe.

Related

If you liked this, you might also like Dan North's presentations Accelerating Agile: hyper-performing teams without the hype and Patterns of Effective Teams at NDC Oslo 2013.