It always starts out the same way — a few machines, a few users, and a few ssh keys to manage. You manually copy their keys (and your own) to the appropriate machines. You might even say “screw it” and share one key for root access. Admit it. It’s ok. Well, no, it’s not ok, but we all do it at some point, even though we know better. Like pouring grease down the drain, we know we shouldn’t do it but sometimes it’s just so much more expedient.
Anyway, you’re lucky enough that your company starts to scale and things start to get more complicated. More users, more servers, tons more keys. Like having too many kids at a birthday party, you start spending all your time making sure they have ice-cream, they don’t fall off the trampoline, they don’t get into fights. You’re spending way too much time managing your users and their keys, so you realize that you need some sort of system to keep everything coordinated. Often this starts off as an Excel spreadsheet tracking who has what key on what machine (another thing we all do but don’t like to admit to our friends). Ok, that’s a start, but that’s just one of many things that you need to manage with your ssh keys, and the less manual the process the better
Centralized store of identities
First, let’s admit that Excel spreadsheets really aren’t sufficient to track who can access what on your servers. Without centralized key management, you’re likely to have temporary accounts that are forgotten and hang around forever. You’ve also got the problem of rogue users… one of the users you authorized users goes around you to add someone else, and you as administrator have no idea. You want to have one source of identity for these machines — one place to add, one place to delete, one place to audit.
Provisioning new users
If I’m adding a user a month — sure, I can go and add peoples’ keys to each machine manually. Of course, this presupposes that you’ve got the public key from the user and have it to move around. Which presupposes they’re savvy enough to manage making keys themselves. A key management system should make this process simpler for you, reading from your identity store and putting them onto the appropriate machines
Deprovision departing users
If you have someone leaving on good terms, it’s just good practice to remove their keys from your servers. Keeps things clean and secure. If they’re leaving on not-so-good terms, though… you better know where the keys are, on every machine, and be able to remove them quickly. This also reiterates the need for keeping separate keys for all of your users. If they have the private key for root, you’re going to have to not only get that key off the machines, but also hand out the new key to everyone that is supposed to have access to the shared account.
SSH Key rotation
Keeping the same key around makes you more vulnerable to attack if an old key is compromised. The PCI DSS standard says that keys must be rotated at least annually. Making this as easy as possible for the user is important, and you must have some place to record when each key was last rotated. For a lot of organizations this is probably that Excel spreadsheet somewhere that’s dusted off right before an audit. Doing this right — rotating all your keys across all your machines on a regular schedule — calls out for an automated process.
SSH Key robustness
Let’s say everyone in your organization is using 1024 bit keys and suddenly word comes down that you need 2048 bit keys instead… having a key management system is going to help coordinate the update. Otherwise you’re going to be looking at every key on every server (notice a pattern here?) to see which need updating.
Getting rid of defunct keys
This isn’t a one time thing — keys (and users) have a tendency to be provisioned “temporarily” and then forgotten and left on a machine. A good key management system will find and remove these keys, or at least notify you of their existence.
Allow users to self-service
Although we don’t want to allow users to willy-nilly add accounts for themselves and each other, we also don’t want to have to create and handle their private/public key pairs and go into every machine and make changes whenever they lose a key, or want to update their keys. Ideally our key management system would allow for (secure) user control of their keys, allowing resets and updates.
While most of these things are possible to do by hand… it’s just not easy and it will end up taking way too much time. Maybe you can script it, and in fact, many companies do. But many companies used to write their own string libraries too, and run their own mail servers, and manage their own payroll. It’s not the best use of your time.
In the next article we’ll explore some of the options out there for managing ssh keys without having to build everything ourselves.