DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Software Design and Architecture
  3. Cloud Architecture
  4. 4 Challenges In Kubernetes Log Transport

4 Challenges In Kubernetes Log Transport

If you are working with Kubernetes for logging, take a look at some of the issues and challenges that you might face that differ from VMs or bare metal environments.

Denny Zhang user avatar by
Denny Zhang
·
Oct. 07, 18 · Analysis
Like (2)
Save
Tweet
Share
7.14K Views

Join the DZone community and get the full member experience.

Join For Free

For the past three months, I have been working on PKS observability features. Right now, it's mostly about Kubernetes logging.

Collect logs, and send them to the log server. That looks quite straightforward. Simple and common, isn't it? Agreed, but only partially. I have noticed some new challenges in the container logging, compared to VM or bare metal environments.

Here is the summary. Check it out! See how much it may apply to your Kubernetes projects. (BTW, our PKS project is hiring).

The motivation for this post is more about illustrating the problems and technical difficulties. Not about how to solve them. And if it has anything against the company's policy, changes will be made on-demand.

Normally the log transport workflow is either active or passive.

Active Way: process actively sends log messages to a remote syslog server. And usually, the format of data encoding is rfc5424.

Image title


Passive Way: for each process, specify the log paths or file patterns. Log agent periodically scans them and send the captured log messages to the log server.

Image title


So you may think the problem has been solved. Not yet, my friends.

Running service in containers are different from VMs or bare metal. New trends are:

  1. The process will be more ephemeral.
  2. Process deployment will be more distributed.

What does that mean to the container logging?

Challenge I: Fail To Collect All Critical Logs

When something is wrong, pods may get deleted or recreated quickly. Consequently, the log file associated with that pod/container will be deleted/created quickly.

However, log agent like Fluentd or Logstash detects new log file by scanning the folder or log pattern periodically. And the default scan interval is 60 seconds(see below figure). The scan interval may be too slow to capture the short-lived pods. How about we set the interval to shorter, say 1 second? The performance overhead would be much higher.

Image title

Previously this won't be a problem in VM world. When the process gets restarted somehow, log file may be rotated but won't be deleted. So users may experience slowness for receiving logs. But not like this: missing critical log for problematic processes.

How we can solve this? Not sure about the best practice, since here in PKS we are also exploring. Maybe we can start a Kubernetes controller subscribing to pod events. Whenever a pod creation event has been fired, notify log agent immediately. honeycomb-kubernetes-agent is an interesting GitHub repo implementing this idea. Please leave us comments, if you have a better solution.

  • Not all logs are redirected to stdout/stderr. If process inside pod writes log to local file instead of stdout/stderr, log agent won’t get it.

Why? It only monitors the log file associated with the pod, like below. And that log file will only capture container's stdout/stderr.


# ls -1  /var/lib/docker/containers/*/*-json.log
ls -1  /var/lib/docker/containers/*/*-json.log
/var/lib/docker/containers/0470.../0470...-json.log
/var/lib/docker/containers/0645.../0645...-json.log
/var/lib/docker/containers/12d2.../12d2...-json.log
...
...


Yes, this logging behavior is anti-pattern for Kubernetes world. However cloud-native movement definitely takes time, not everyone is fashion enough. This is especially true for DB services.

Compared to VM worlds, Pod may move across different worker nodes quite often. But you don't want whenever K8s cluster has one pod change, the log agent needs to be reloaded or restarted. New challenges, right?

Challenge II: Multi-Tenancy For Namespace Logging

Kubernetes workloads are usually running in shared worker VMs. Workloads from different projects are divided by namespaces.

Different projects may have been different preferences for logging. Where the log goes to, and managed by what tools, etc. Need to provide an easy way to configure and with no extra security compromises.

It turns out Kubernetes CRD (CustomResourceDefinition) is a good fit.

  1. All you need to learn is the standard  kubectl command . (See kubectl cheatsheet).
  2. RBAC can be applied to this custom resource. So security can be easily enforced.

In PKS, we call this feature as sink resource. Note: this idea has been proposed to Kubernetes community. Hopeful it will be merged into upstream soon.

Image title

Challenge III: Support Logging SLA For Different Namespaces

For simplicity, people usually only deploy one log agent as Kubernetes daemonset. It means one pod per Kubernetes worker node. If somehow this pod needs to be reloaded or rescheduled, it will impact all Pods living in this worker node.

Starting from K8s v1.12, each node may run 100 pods. You need to make sure your log agent is fast enough to collect logs from all the pods.

Like any shared environments, you may experience a noisy neighborhood issue. The misbehavior of one Pod will hurt all other pods in the same worker node. Want to disable logging for one problematic namespace? You can easily avoid emitting the log, but not the part of collecting log.

A slow disk may create significant latency for log transport. Failure to handle back-pressure issues may DDoS your log agent.

Challenge IV: Handle Logging From Different Layers

Like the below figure, we have pod logs, K8s logs, and platform logs. Even for "pod logs," we have logs from standard workload or from K8s add-ons.

As you may guess, different types of logs have different characteristics. And they may have different priorities. Not only layer vs. layer, but also different SLA for the same layer.

To provide a K8s solution, how we can address this? Facilitate Ops/Dev to find out the root cause quickly. Meanwhile, minimize the security compromises.

Image title

What is PKS? PKS is an enterprise Kubernetes solution from VMware and Pivotal.

Kubernetes pods

Published at DZone with permission of Denny Zhang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Why It Is Important To Have an Ownership as a DevOps Engineer
  • Handling Virtual Threads
  • Integration: Data, Security, Challenges, and Best Solutions
  • How To Generate Code Coverage Report Using JaCoCo-Maven Plugin

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: