How I Discovered My Security Practices Are Pretty Bad

I have a confession to make. As a software developer, I never thought much about security. It was about as important to me as the daily weather on Pluto. Admittedly, my work was mostly deployed in lab environments, but I never did much to look for CVEs, attack paths, and other security vulnerabilities.

As a developer, I made some presumptions:

• The latest Linux images I use (Ubuntu, AlmaLinux, openSUSE, Alpine) are secure by default.
• If deploying to AWS or another public cloud, the cloud provider took care of the security I cared about.
• The infrastructure-as-code I used to provision systems was simple and therefore solid.
• I’ve been safe from attacks for years, so the risk was low.

My presumptions were wrong on all counts.

Linux Images Aren’t All That Secure Out of the Box

When I worked for Chef, the software-defined infrastructure folks, I used a tool called Inspec to quickly analyze systems running the latest (and updated) versions of Linux OS via ssh. I used publicly available benchmarks — like this Linux one from DevSec — to scan for problems. I was stunned by the results. They showed 60 of the 177 vulnerability tests failed on my simple mail server running Ubuntu:

$ inspec exec https://github.com/dev-sec/linux-baseline -t ssh://myuser@mail.tiny.lab -i ~/.ssh/id_rsa
…
Profile Summary: 28 successful controls, 29 control failures, 1 control skipped
Test Summary: 115 successful, 60 failures, 2 skipped

The errors include routing problems, lax file and directory permissions, and CPU and file-system vulnerabilities. So much for base Linux images being secure out of the box! I now realize these base images (whether I use .iso files or Docker images) are designed to provide modest protections, but not all. And the older the Linux distro version, the worse it gets.

I found this to be true regardless of whether I tested systems in my home lab or instances running in AWS. The base images are not automatically secure, and my sample was only testing using a Linux baseline profile. Running more specific tests — like the github.com/dev-sec/ssh-baseline — showed 44 successful tests and 60 failures!

My Cloud Instances Were Even Less Secure

Similar problems revealed themselves when I scanned instances in AWS using InSpec and Tenable’s Nessus and Cloud Security tools. Nessus uses an agent installed on each instance to grab and report on vulnerability data, and Cloud Security uses an agentless approach that looks at all instances running in my VPCs. The latter revealed misconfigurations and vulnerabilities in my instances and with the cloud environment itself, such as IaM role problems, subnet weaknesses, public IP addresses, and risks associated with using my default VPC for instances.

Again, I was stunned by the sheer number of vulnerabilities and misconfigurations I had in my running systems, including many problems marked “critical.” Perhaps most troubling was the fact that these security risks are the result of my oft-used base Terraform code settings. Admittedly, I’m not an AWS or Terraform expert, but I figured I knew my way around pretty well and was following safe practices. In a word, the reality was more like “sorta.”

Committing to Fixing My Errant Ways

The Tenable tools I used offer instructions on how to fix most vulnerabilities, and I immediately went into my AWS dashboard to fix my Security Groups and IaM roles to make them more secure. I took a fresh look at the code I use to provision systems, too, to make sure I was following good security practices.

With those changes in place I went on to set up an AWS site-to-site VPN, which allows me to access my instances using private AWS IP addresses, not public ones. That way, I can access the nodes without ever having to attach risky public IPs. Since my nodes are primarily for testing — and not public-facing — this was a reasonable added bit of security.

To be clear, most of my work is in development and testing and nothing I stand up contains proprietary data. Still, I was careless. Perhaps even more so because I was mostly just “testing” and not running production systems. The problem with that presumption — particularly for the things I spun up in the cloud — is that poor cloud settings left open ways for hackers to get access to my AWS infrastructure, not just the instances themselves. AWS covers a lot of the security bases, yes, but my actions (or inactions) quickly and easily put me at risk.

Going forward, I’ve hardened my Terraform code by adding much more specific definitions for my VPCs, subnets, roles, and other cloud features. I use tools to regularly check for and apply patches to my instances, even if they’re running the latest OS version. I’m more deliberate in how I create and destroy machines, and I’ve started relying more on Terraform Vault for secrets management and Tenable and Terrascan for regular code and instance scanning. 

Though I’m not much of a New Year’s resolution guy, taking security more seriously is one commitment I plan to keep in 2023 and beyond.