Welcome to Distributed

This year’s Gluecon as always was a mix ideas and memes (a great event as always!). Reflecting on the talks and conversations on the flight back, there seemed to be one to us which stood our more than any other, which was nicely captured by Michael Rose‘s Tweet:

"We're all distributed systems people now." #gluecon @argv0

— Michael Rose (@Xorlev) May 22, 2013

In fact, we always were, but conveniently forget that fact most of the time. Specifically, APIs, Cloud deployments, SAAS and “remote access to everything” make building systems incredibly hard because they are distributed – across organisations. The fact that building systems in which multiple components that we depend on are hosted in remote infrastructure owned and operated by others has been a reality for a good long while. However, the increasing adoption of APIs in particular, simply makes the issues that arise more acute – and we don’t talk about the enough. The theme came out strongly both in John Sheehan‘s Keynote on Runscope:

"Distributed is the new Default" for modern applications - "we're building distributed systems by not acting like it" @johnsheehan#gluecon

— Steven Willmott (@njyx) May 22, 2013

and in Andy Gross‘s:

GlueCon 2013 Notes: The Free Lunch Is Over, Again – Andy Gross, Chief Architect at Basho bit.ly/16Rt7vg

— Tony Karrer (@StartUpRoar) May 23, 2013

John’s talk covered the pain of working with multiple different external APIs – both at development time and at run-time. Each API fails in different ways and at different times – making even gaining insights as to what might be wrong hard to glean, particular with systems that need to run 24/7. Andy dug right in on the kind of testing needed to really validate that a distributed system such as the deployments Basho – “Concurrency is no longer enough – We are distributed Systems people now”.

The theme also came out in Talks from Netflix, Google and Parse – not only do internal systems (which may be distributed) need to be managed closely: there are now external systems which depend on these and others on these.

In a “distributed” world many things become harder: remote nodes cannot always be relied upon, clocks are not synchronized, small latencies between locations make many algorithms invalid or impossibly slow, many of the endpoints are managed by different organisations – meaning changes are often not coordinated or orchestrated.

All this warms ours hearts as an “old time” distributed systems folks – Internet Applications are becoming genuinely distributed across organisations. This creates growing pains – but also makes new types of application possible. People often ask “what’s the fundamental difference between SOA and APIs? The essence of the answer is, not technical – it is organisational:

APIs are typically designed for inter-organizational integration, SOA typically for intra-organizational.

Even though SOA solutions do get used for the former, this has almost always happened with strong assumptions on the human collaboration around the integration and sense of “shared ownership” of the resulting app – control of the resources involved. With the rapidly increasing number of API integration, this assumption is simply not there. Apps in an API world need to handle being truly distributed.

Vendors like ourselves and new companies such as Runscope, will hopefully start to make it easier for both sides of the API equation to take some of the pain out of providing and consuming APIs. Building genuinely functioning, stable apps across clouds is very hard and we have a long way to go – but we’re starting to recognize and address the pain points!

Huge congrats to Eric, Kim and the team for another great event!