The recent problems with NPM and Kik have really shined a light on the poor state of supply chain management in our industry. This is a case where the involved actors, acting out of legal obligation and understandable self-interest caused a cascading failure across multitudes of seemingly unrelated systems. Emergent behavior indeed.
To recap, for those of you who have been stranded on Mars or trapped in a basement somewhere, here’s basically what went down — so there’s this instant messaging app named Kik, a prolific open-source developer named Azer Koçulu, and NPM, the package manager we all know and love and the company that supports it. So Azer created a module named Kik within NPM for some work he’d done. Kik (the company) caught wind of this, and asked Azer to move his stuff to a new module name (they need to do this kind of thing, if you don’t protect trademarks, you lose them). Azer said he wouldn’t do it, Kik brought out the lawyers, and in response Azer unpublished everything he’d ever developed. From Azer:
This situation made me realize that NPM is someone’s private land where corporate is more powerful than the people, and I do open source because Power To The People.
Personally, I think that all parties involved were acting in accordance with their own principles and self interest, and I don’t have a problem with that. Azer created everything he published, and had copyright over it. If he wants to unpublish any or all of it, I feel he’s well within his rights to do so. Likewise, Kik didn’t really have a choice — they had to defend their trademark, or they’d lose it. I do think that there’s some potential for discussion around NPM, the company, and NPM, the repository, with respect to expectations for published code, but I think that’s secondary to the original conflict, to be honest.
So here, we have Azer and Kik, two parties on opposite sides of the globe acting in their own understandable self interest and bringing the internet down as a result. Yikes.
What went wrong?
Supply Chains and Trust Models
There’s lots of places where this could have been stopped. NPM could have followed an immutable publication policy, for example. Companies using NPM could have used a library caching strategy. Perhaps Kik could have given Azer some kind of exemption (I don’t know if this is really an option, I’m not a lawyer. Also, really happy about that). From my perspective though, the key problem lies in the supply chain companies and other projects used, and the trust implicit in that supply chain.
Most of us don’t think about supply chains much. It’s boring. And who cares, right? This kind of stuff doesn’t impact us. And that kind of thinking exists in many, many software projects as well. Open source libraries have a zero initial acquisition cost, after all, for most of us (which is bogus, by the way — if companies use FOSS software, they should contribute and support those projects. But that’s a rant for another time). So people don’t spend any time optimizing FOSS supply chains — there’s no money in it, right?
Well, wrong. Time is money after all, and when Azer pulled left-pad, lots of software crashed. So maybe we should pay attention to this supply chain stuff after all.
How Supply Chains Work
Companies that have been able to really optimize their supply chains have done it by being so dominant that they’ve been able to strong-arm their suppliers to do things like install compatible software systems and optimize their own supply chains. Think companies like Target, or WalMart, if you’re in the US. If you don’t, I’m sure you have equivalent retail behemoths in whatever country you live in too.
In the NPMGate case, our supply chain is really a transitive dependency tree. And to make it even more exciting, the dependencies span multiple projects, with different maintainers, with undocumented (or unknown) dependencies. In this kind of environment, it’s not even clear we can known the supply chain. Unless, of course, you use a dependency manager like, you know, NPM.
So the package.json file contains a nifty section that describes dependencies. This allows tools to traverse dependency trees and load needed software. If each dependent project also contains a package.json file, you’re able to track every transitive dependency too. So really, in this case, we can know the complete dependency tree. NPM (the tool) does this - you can too.
The problem is, you didn’t. That’s okay, really, nobody did.
Why? Well, when you don’t spend any time optimizing supply chains, you do it because they have very little value. And if they have very little value, why spend any time on it? Makes sense. It’s just wrong. And to make things worse, supply chain issues are low frequency, high consequence events, things us human-brand animals are really, really bad at managing (see: your retirement savings). So now we know that we can’t trust our supply chains - those high consequence events just became real. So what do we do?
What You Can Do
Well, stop implicitly trusting them. Rather, you need to explicitly distrust them. And you need to know what they are, or you need to be able to extract this kind of information. Your best bet is to use your own NPM repository that caches the software you use (see: Sinopia, npm-proxy-cache, or npm_lazy for potential options). While you still implicitly trust the functionality of the delivered modules, you can still deliver working software if you have routing problems, slow networks, or unpublishing issues. More sophisticated approaches might run all module unit tests and check unit test coverage too, allowing some visibility into module functionality.
Bottom line, we all need to grow up. It’s not Kik’s fault that your site crashed. It’s not Azer’s or NPMs either, really. It’s yours. Now go out there and do something about it.