Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Kubernetes Security Best Practices

DZone's Guide to

Kubernetes Security Best Practices

Keeping your Kubernetes clusters and containers secure is key. We take a look at some best practices for you to follow in order to help keep your instances safe.

· Security Zone ·
Free Resource

Protect your applications against today's increasingly sophisticated threat landscape.

Kubernetes (K8S) is an open-source container orchestration tool that can automatically scale, distribute, and handle faults on containers. Originally created by Google and donated to the Cloud Native Computing Foundation, Kubernetes is widely used in production environments to handle Docker containers (although it supports other containers tools such as rkt) in a fault-tolerant manner.

Security should be a top priority for any production system and must be even stricter when securing clusters since they involve more moving parts that need to cooperate with one another. Securing a simple system involves maintaining good practices and updated dependencies, but to secure an environment, whether clustered or not, the operator needs to evaluate the communications, images, operational system, and hardware issues. Data breaches, Denial of Service attacks, stolen sensitive information or simply downtime, can all be avoided with solid security policies.

As an open-source system for automating the deployment, scaling, and management of containerized applications, Kubernetes impacts many runtime security functions. As with any open source project, issues are rapidly discovered but any user must keep their software updated to avoid opportunistic attacks.

In the following sections, we'll take a deep dive into some security practices that will help you avoid issues when deploying your own Kubernetes instance.

Cluster Security Overview

Clusters are a group of servers working together as a single system. As such, they are complex, need to be constantly updated and monitored and, as with any distributed system, can be more prone to failure. In addition to the typical security issues that involve any computer software, such as bad programming and out-of-date dependencies, clusters have their own specific security pitfalls. For example, a bad network configuration can expose the entire computing system to an unauthorized user; a single node with an outdated operating system can lead to a breach of all your machines; a system subjected to a DoS attack could lead to one or more machines being rendered unusable.

In this case, one of the most important tasks assigned to system administrators is to ensure the security of cluster installations while, at the same time, permitting easy access to legitimate users. We should divide the topic into three areas: the cluster, as a set of (virtual) machines that should be orchestrated and maintained, the container's structure and its communications, and the applications inside those containers.

Kubernetes Security Practices

Kubernetes is the world's most popular container orchestration tool and is here to stay. In 2017, it rose to dominance because of features, community, offerings in the cloud, and recognition by its competitors like Docker and DC/OS.

Still, in the Kubernetes environment, there are threats that may result in compromises and undesirable scenarios, including an elevation of privileges, exfiltration of sensitive data, a compromise of operations, or a breach of compliance policies. These issues can be mitigated by following strict security practices such as the following:

Environment Separator + Authentication

A cluster can be used for different environments and different purposes: it can have services for several production products and even for a variety of purposes: testing, staging, production, and so on. It is important to separate these into different namespaces, so you can control access to the resources the service has access to. Namespaces create a network layer with resources within the same space.

Production environments should always be in a separate cluster with strict access permissions. Nevertheless, for other environments, it is possible to create roles for each namespace so only your QA team can access the testing environment.

The APIs are the central interfaces for administrators, users, and applications to operate and communicate in the Kubernetes environment. For that reason, controlling API access is the main task of authentication and authorization within Kubernetes.

The Kubernetes platform has built authentication and authorization controls and admission controls, which intercept and regulate requests to the APIs after authentication and authorization.

Keep the Cluster Updated and With Reliable Images

Every type of software contains bugs. Related to security, malicious people around the world are constantly trying to find the next breach in popular software. It is a cat-and-mouse situation as security issues are explored, discovered, and solved while a new one will pop up in a few days. It is the cluster operator's responsibility to keep all software running on the cluster updated so major flaws are repaired before they've been exploited.

Keeping software updated is a hard task that needs to be carried out in the three layers mentioned before (cluster, containers, and software).

  • Cluster - The operator should always monitor the K8S version and the SO's versions of each node. Maintaining updates via a mailing list or announcements of the software in use should avoid the majority of known security issues.
  • Containers - Docker images are made of layers that increase the complexity of a container. For example, if you have a Java Web Application, your image might have the following layers: [Distribution Layer], [JRE Layer], [Tomcat Layer], [Your application layer]. Each layer could have a security flaw and must be updated. Discovering these issues can be a tedious and error-prone task. Fortunately, there are automated tools to discover those flaws like Sonatype Nexus, CoreOS Clair, and Dockscan.
  • Software - Keeping your own software updated means keeping your dependencies updated and avoiding common development errors (SQL injection, for example). This step needs to be carried out in the Continuous Integration phase as this is the moment the software can have its source and dependencies exploited. Sonarqube is a tool that analyzes your code and provides insights about code quality and security issues. You should also add a platform-dependent tool to check your dependencies and find known security issues.

Keep in mind that, if a container is defective, Kubernetes will exacerbate the problem by deploying it across a number of machines.

Most organizations use open-source components to build applications and they also use prebuilt images for their containers. Once you pull down images from a public hub, which are not validated and signed, you create an open space for untrusted content.

So, in order to protect your container, we recommend using private or official registries, maintaining standard base images for developers to use, and scanning images for security vulnerabilities.

Resource Quotas

It is important to define quotas for your resources. Unbounded resources can lead to total cluster unavailability in the case of DoS attacks or malfunctioning applications. If a resource is not bounded, it can draw all the available hardware resources to itself.

Kubernetes has several resource quota configurations available. It is possible to (1) define the maximum number of instances of the same container, (2) the number of CPU shares the application will be able to consume, and (3) the maximum amount of memory. These configurations can be defined for a POD or to a namespace.

Network Segmentation

Though namespaces offer DNS-level network segmentation, you would need a firewall to block undesirable incoming and outcoming network communications. Kubernetes offers a simple way to protect the communication between pods with Network Policies.

With this configuration, an operator can set which pods can communicate with other, at which port, and in which direction. Here is an example:

{

  "kind": "NetworkPolicy",

  "metadata": {

    "name": "pol1"

  },

  "spec": {

    "allowIncoming": {

  "from": [{ "pods": { "segment": "frontend" } }],
  "toPorts": [{ "port": 80, "protocol": "TCP" }]
    },
    "podSelector": { "segment": "backend" }
  }
}

Logging is not only used to discover bugs in systems, it can be used to discover bad behavior and unexpected usage in the system. Monitoring the system log using the ELK Stack provides several insights that would help an operator to discover these problems.

Again, we can separate logging into three areas:

  • Cluster - For cluster monitoring, we should observe system health and how all the nodes are behaving. Different historical usage without any other new facts could indicate a security issue that should be analyzed and addressed.
  • Containers - There are several metrics provided by K8S itself per POD: how each pod is using its resources (CPU, Network, memory), how many instances it is using and if the deployment of new versions are healthy or not, for example. Any drastic change when comparing past results could indicate an issue and should be investigated.
  • Application - Finally, application logs could lead to their own new set of metrics like the number of database connections, the number of requests, the request duration, etc. Any drastic changes in these numbers can help the operator to identify security issues over time.

Over time, using logs to monitor your cluster will help to identify changes in applications behavior that could lead your system open to attack. The key factor here is you need to store historical data to be able to compare different periods. There are several reasons, both good and bad, that these changes may have been made but they should be identified by a human operator or machine learning algorithms.

Conclusion

Securing a system is challenging, but securing a cluster is an even more daunting task. In order to do so successfully, the operator needs to understand the behavior of an application where hundreds of machines are working together.

An operator should consider the software versions running on the cluster, monitor various aspects of all the applications, and prevent unauthorized access to the cluster. Moreover, one should consider how the application normally behaves and identify odd behavior that could help identify a security breach.

Cluster administration is a complex endeavor. In order to maintain security, everything has to be constantly monitored, kept updated, well separated, and with good authentication roles. Through diligence and attention to these points, your cluster will be safe and sound.

Rapidly detect security vulnerabilities in your web, mobile and desktop applications with IBM Application Security on Cloud. Register Now

Topics:
kubernetes ,security ,container security ,devsecops

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}