Security With Falco

Introduction

Securing a container platform is a multi-step process spanning from development to production. The process of securing containers is continuous. It should be integrated into your development process, automated to remove the number of manual touchpoints, and extended into the maintenance and operation of the underlying infrastructure. Container security thus is the process of implementing security tools and policies to ensure that your container is running as intended.  

There are a lot of Docker security practices and Kubernetes security practices. But there are gaps: there could be Image vulnerabilities or container abnormalities still. Some of these can be captured and addressed using various static container image scanning tools. But there is a need to analyze the container behavior at run-time to detect any bogus configurations, which are intentional or not, leading to data loss, security intrusions, and eventually leading to different vulnerabilities. If a container is not running as expected, then it could attack that exploited an existing vulnerability. So the run-time container scanning is essential to detect any action that deviates from the norm.

There are a lot of exposed Kubernetes dashboards or exposed Kubelet APIs because people are opening every port to the internet.

“Runtime security is a critical piece in a cloud-native security story, said Kris Nova, Sysdig’s chief open source advocate. “Access control and policy enforcement are important prevention techniques, but runtime security is needed to detect threats that evade preventions.” 

Falco

Falco (created originally by Sysdig company) is an open-source run-time behavioral activity monitor for container-native infrastructure. Its a CNCF incubator project and Falco software have become the de-facto open-source standard for cloud-native runtime security.  Falco focuses on behavior monitoring. 

Falco gains visibility inside the containers through system call instrumentation, with great performance and very little overhead, so can run in production systems. System call instrumentation is completely transparent to the containers, so there is no need to modify the code, container images, or inject or preload libraries. System call interception is done either through a simple kernel module dynamically compiled with DKMS or through an eBPF probe.

When an anomalous activity is detected, a security event is logged and alert is triggered based on the rules and policies that we can define in the software (similar to tcpdump and sysdig rules). We can write custom rules also as needed. Falco is container-native, so rules and alerts are going to understand what is a process but also a container or a Kubernetes pod.

Problems That Falco Detects

1. Are the hosts or containers doing something that they should not?

2. Are there any spawned or unexpected processes running in the container?

3. Any changes to the running container configuration detected?

4. Any new ports opened or any unexpected network activity happened?

5. Does any user spawn a shell into the container?

Falco Architecture

Falco can detect and alert on any behavior that involves making Linux system calls. Falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process.  Falco operates at the user space and kernel space. The system calls are interpreted by the Falco kernel module. The syscalls are then analyzed using the sysdig libraries in the userspace. The events are then filtered using a rules engine where the Falco rules expressions are configured. Suspicious events are then alerted to an output mechanism, configured as Syslog, files, or stdout solutions. 

Falco Detection Workflow in Kubernetes

Falco also helps to capture K8 audit events. K8s audit events provide a chronological set of records documenting changes to cluster and each record is a JSON object. The detection workflow is the same as explained in the earlier section. 

Falco Runtime Monitoring Solution in K8 Cluster

Falco can be integrated with an external monitoring solution like EFK or ELK. For Demo purposes, implemented below solution in a Kubernetes cluster on AWS. 

Steps to implement:

1. Bring up an AWS cluster with one master and two nodes. 

2. Falco was installed directly on each of the host systems using the below commands. 

xxxxxxxxxx

curl -s https://falco.org/repo/falcosecurity-3672BA8F.asc | apt-key add - echo "deb https://dl.bintray.com/falcosecurity/deb stable main" | tee -a /etc/apt/sources.list.d/falcosecurity.list apt-get update -y

xxxxxxxxxx

apt-get -y install linux-headers-$(uname -r)

xxxxxxxxxx

apt-get install -y falco

2. Falco runs as a service on each of the nodes.

xxxxxxxxxx

 systemctl start falco

3. Once Falco services are running, we have to change the /etc/Falco/Falco.YAML file, and enable jso output to true. Also, configure the file path where Falco logs will be captured.  

xxxxxxxxxx

# Whether to output events in json or text

json_output: true

file_output:

  enabled: true

  keep_alive: false

  filename: /etc/falco/falco.txt

4. Restart Falco daemon on the nodes. 

5. Falco will start monitoring and redirect the logs to /etc/falco/falco.txt

6. Install filebeats and ELK stack using helm charts. 

7. In the Filebeats YAML file, change to include Falco log output path. 

8. Open Kibana and you can configure different reports from Falco configured as:

 >>> Table of Falco alerts by Kubernetes Node

• Split Rows
• Aggregation: Significant Terms
• Field: Kubernetes.host.keyword
• Count: 20

 >>> Pie chart of top 10 Falco rules triggered.

• Split Slices
• Aggregation: Significant Terms
• Field: rule.keyword
• Count: 10 

>>> Area chart of all Falco alerts over time.

• Aggregation: Date Histogram
• Field: @timestamp
• Interval: Minute

Some of the error logs captured are:

xxxxxxxxxx

{"output":"13:35:55.104817000: Notice Privileged container started 

{"output":"13:37:18.578581433: Error File below /etc opened for writing

{"output":"13:36:21.084924818: Notice Namespace change (setns) by unexpected program 

{"output":"13:37:24.618659538: Error File below a known binary directory opened for writing 

{"output":"13:37:27.714296811: Notice Unexpected setuid call by non-sudo, non-root program 

{"output":"13:37:41.848085676: Warning Sensitive file opened for reading by trusted program after startup 

{"output":"13:37:50.994157670: Error Rpm database opened for writing by a non-rpm program 

{"output":"13:37:21.617518572: Notice System user ran an interactive command 

NOTE:  The above is the most secure way of running Falco. We can also install Falco using a helm chart. Falco install using Helm is generally as follows :

xxxxxxxxxx

 helm repo add falcosecurity https://falcosecurity.github.io/charts

xxxxxxxxxx

helm repo update

xxxxxxxxxx

helm install falco falcosecurity/falco

 This will result in a Falco Pod being deployed to each node (including master), and thus the ability to monitor any running containers for abnormal behavior.

Falco agents will be spinning up on each node in the cluster. After a few seconds, they will start monitoring the containers looking for security issues. 

Configuration Inputs (Custom Files)

 The values.yaml that have the parameter values to be set during the Falco install.  The JSON output and the file log path can be changed in the values.yaml. 

xxxxxxxxxx

helm install falco -f values.yaml falcosecurity/falco

This chart adds Falco to all nodes in the cluster using a DaemonSet. All the alerts will be to Syslog and the same can be integrated with ELK stack to capture events on the kibana dashboard.  

Falco and Kubernetes Security Stack

While detecting anomalous behavior in your containers is step one, and step two is responding to or mitigating the attack in an automated way. In the above automation, Falco evaluates the rules and triggers alerts. A small forwarder in a container within the same pod reads the alerts through a named pipe and then sends them to NATS using TLS.  NATS is a messaging broker so other parties can consume security events.

• Falco – agent that detects the wrong behavior
• NATS – Open Source Messaging system, a response engine that routes the messages to an endpoint
• Playbooks or functions — to mitigate those issues caught and respond to the security incident. A Function is an independent unit of deployment, like a microservice. It's merely code, deployed in the cloud, that is most often written to perform a single job 

Conclusion

Falco is extremely effective in part because its rules can use context from the low-level operating system interface (syscalls) together with Docker container or Kubernetes orchestration metadata (container information, namespaces, deployment, pods, etc). Any reaction to a security incident needs to be fast. Because containers have short lifespans, it’s essential to find the culprit before the container causes any real damage or simply disappears or moves somewhere else. Automation is more important than ever, and writing response procedures as code playbooks is the best approach.