Securing Your Infrastructure and Services During the Distribution Phase

In the previous article, we discussed how to incorporate security during the design phase of the software development life cycle. Some of the strategies included threat modeling, static analysis, and code reviews. Now it is time to move past the design phase to the distribution phase.

The distribution phase involves three different steps again.

• Static Analysis
• Image and Manifest Scan
• Signing

We already discussed static analysis so we will directly go image scan and signing.

Image Scanning

One main criteria for services to run and scale in the cloud is to deploy them in containers, and containers are created from images. The images can in turn inherent from other base images and define dependencies. To keep our services secure, it is important to address vulnerabilities coming from all these places. For example: 

# Use official Python image as a base
FROM python:3.11-slim

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1

# Set work directory
WORKDIR /app

# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy project files
COPY . .

# Run the app
CMD ["python", "app.py"]

In the Dockerfile above, the base image comes from a Python base image. Then we have a requirements.txt with a set of packages installed using the pip command. So how do we make sure this base image and its dependencies being installed do not have critical or high vulnerabilities? We could use open-source tools like Trivy or Clair that can perform a scan and notify us about possible vulnerabilities.

The output we see is after running the trivy.exe image scan on the container. 

 docker build -f .\Dockerfile.dockerfile -t my-app:latest .

./trivy.exe image myapp:latest

In total we have 1 high and critical vulnerability and 28 medium vulnerabilities. Normally, critical vulnerabilities are the ones that need immediate attention. You could configure your build to fail when there are serious vulnerabilities because they could be exploited to infiltrate the services.

Another best practice to maintain low vulnerability count is to limit the number of available base images. Simply put, minimal base image recommends only install required dependencies and reduce the number of layers. As a result, you have fewer dependencies to deal with and less churn to impact your development lifecycle.

Manifest Scan

With the raise of Kubernetes workloads, the configurations are defined in a manifest file. The configuration in the file acts as a source of truth for the Kubernetes control plane when it has to bring up the cluster or pods. 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: nginx:1.21
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

The above manifest file describes:

Deployment: 

• Deploys 3 replicas of the Nginx container
• Uses the nginx:1.21 image 
• Exposes port 80 inside the pod

Service:

• Selects pods labeled app: nginx
• Exposes port 80 and forwards it to the pod's port 80 
• Uses LoadBalancer to expose it externally

Running services as root without setting resource limits is considered dangerous as a DDoS attack will drastically increase the container count and drive your infrastructure costs up. To identify such vulnerabilities we could run a Trivy scan on the manifest file we previously discussed.

.\trivy.exe config .\k8.yaml

Each vulnerability shows an identifier, a description, and where in the file the vulnerability is coming from. For example, AVD-KSV-0012 shown below talks about specifying readOnlyRootFilesystem flag to true to make sure nobody is able to write to the underlying file system.

AVD-KSV-0012 (MEDIUM): Container 'nginx' of Deployment 'nginx-deployment' should set 
'securityContext.runAsNonRoot' to true
════════════════════════════════════════
Force the running image to run as a non-root user to ensure least privileges.

See https://avd.aquasec.com/misconfig/ksv012
────────────────────────────────────────
 k8.yaml:16-19
────────────────────────────────────────
  16 ┌         - name: nginx
  17 │           image: nginx:1.21
  18 │           ports:
  19 └             - containerPort: 80
────────────────────────────────────────


AVD-KSV-0014 (HIGH): Container 'nginx' of Deployment 'nginx-deployment' 
should set 'securityContext.readOnlyRootFilesystem' to true
════════════════════════════════════════
An immutable root file system prevents applications from writing to their local disk.
This can limit intrusions, as attackers will not be able to tamper with the 
file system or write foreign executables to disk.

See https://avd.aquasec.com/misconfig/ksv014
────────────────────────────────────────
 k8.yaml:16-19
────────────────────────────────────────
  16 ┌         - name: nginx
  17 │           image: nginx:1.21
  18 │           ports:
  19 └             - containerPort: 80
────────────────────────────────────────


AVD-KSV-0015 (LOW): Container 'nginx' of Deployment 'nginx-deployment' should set 
'resources.requests.cpu'
════════════════════════════════════════
When containers have resource requests specified, the scheduler can make better
decisions about which nodes to place pods on, and how to deal with resource contention.

See https://avd.aquasec.com/misconfig/ksv015
────────────────────────────────────────
 k8.yaml:16-19
────────────────────────────────────────
  16 ┌         - name: nginx
  17 │           image: nginx:1.21
  18 │           ports:
  19 └             - containerPort: 80
────────────────────────────────────────

Scanning is an important phase in the distribution lifecycle as it ensures the artifacts being deployed or released to customers are secure and resilient against common cybersecurity and supply chain attacks.

Artifact Signing

Last but not the least is signing. Signing artifacts serves as a proof of their authenticity, integrity, and trust for components such as:

• Container 
• Images 
• Binaries Manifests (like Helm charts, Kubernetes YAMLs) Packages

When a pod comes up, we can write custom policies to ensure only images that can be verified using their signatures are allowed to run. Cosign and Notary are some open-source tools that can be used to verify the image signature.

cosign sign --key cosign.key myrepo/myapp:latest
cosign verify --key cosign.pub myrepo/myapp:latest

The cosign sign command is used to sign a container image using your private key. Then, consumers of the image can use cosign verify to ensure it was signed by the expected provider.

Conclusion

Scan your artifacts to ensure they are free from vulnerabilities, then sign them to safeguard both you and your customers from supply chain attacks. These are fundamental steps in shipping a secure infrastructure for your customers.