It's day 3 of OSCON 2015 in Portland and two things have struck me so far. The first is that open source keeps getting stronger -- in both adoption and quality. The second is that developers are getting better at injecting human intelligence into their code. Both share a common theme: computing is less and less a constraint on and more and more an augmentation of human intelligence.
Point the First: Open Source Keeps Getting Bigger
Yes, open source can be dangerous. Indeed, enterprises that can't afford to 'fail' at all (let alone fail often) need to take a special kind of care when folding open source components into their proprietary solutions.
But of course open source is big and excellent, and bigger and better than ever. The biggest software companies are pushing it because they're using it to solve their own problems. Allison Randal (HP) observed this morning that 78% of companies are using open source today, up from 48% five years ago. James Pearce (Facebook) explained that sheer intent to open-source internal applications actually makes Facebook write better code -- because nobody wants an open-source application with tight coupling to some random firm (no matter how important) and because the community can be a harsher critic than the most demanding product manager.
Why not? these OSCON speakers are saying, inspiringly and persuasively. Innovation needn't be aimed at immediate profit; rather, profit is a by-product of innovation. In fact, the purest innovation is the simplest response to real need -- not demand manufactured to refecundate a saturated market (to which profit as such is indifferent), but actual absence in the universe of something that absolutely ought to exist because it certainly will make people's lives easier. For any good product, the beneficiaries are the target market, usually consumers or industry. But for much [F]OSS, the beneficiaries are often developers themselves -- smart people solving their own problems by inventing something new, then letting others use the thing they needed to invent for themselves anyway. Kingmaking and open-sourcing go hand-in-hand because the market is efficient enough to reward the creation of real value.
Point the Second: Now We're Better at Using Our Brains to Code
The first observation was a little philosophical along well-trod open-source lines. The second is a bit more practical, a sign that annoying things about computers are beginning to die: a growing interest in making tools that harness more of the developer's mind than his or her abilty to translate requirements into formal syntax.
A genuinely fun talk by Jan Paul Posma today at OSCON inspired this thought. Jan spoke about visualizing program execution -- for learning, as in his jsdares project, but also for deep debugging, as in the fascinating academic (but highly pragmatic) project Whyline. The reasoning behind these visual debugging tools is cognitive-scientific and empirical: humans are drop-dead amazing at finding patterns in visual fields. There are some aesthetic reasons for this, too. A painting is absorbed by the viewer in the order determined by the viewer's eyes, while a poem or a song is received by the auditor in the order determined by the performance.
In more programmatic terms: streams are ordered by the input, execution by the agent. Viewers explore pictures but are enraptured by music. Nobody wants to be enraptured by a program executing unexpectedly. And the frustrating linearity of breakpoints has always made me feel computationally dishonest, an oversimplifying hop over whatever I (for whatever reason) guessed was not the locus of the problem. In-browser web developer tools allow for some nonlinear exploration of the DOM rendering process (and associated network traffic etc.) in a way proper to the visual arts, but rarely does this level of exploratory freedom enter the level of the code itself. Whyline facilitates such exploration, and Jan's talk showed how this might be done in very concrete detail. I'm excited to try these tools myself soon.
The upshot is that we can do much better to use our individual mental abilities to understand what actually happens during execution (without shooting at breakpoints in the dark or glazing over at ten thousand lines of console logs) and to sharpen one another's code by letting the real-world needs of many communities drive the application to new levels of feature-richness and stability. Increment the mental power of the token by adding eyes to letters; increment the mental power of the type by putting more (and more kinds of) eyes on the repos.
Software comes into being in order to meet specific users' needs, but once a codebase exists it can meet needs its creators never dreamed of. Open-source software is an hourglass like metabolism, not a funnel like a SWOT product. Anyone sufficiently intelligent with sufficient resources can achieve a dream; well-supported will can reach any longed-for goal. Only an open set, a community of practice driven by previously unsolved and massively diverse problems, can achieve beyond the original dreamer's dream and drive a product beyond the first makers' intent.
This, anyway, is the basic dream of open source. The vision seems so palpable at moments like OSCON.