DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. Sysdig and ELK: A Match (Potentially) Made in Heaven

Sysdig and ELK: A Match (Potentially) Made in Heaven

Here's a nice tutorial on installing and using Sysdig and sending its information to the ELK stack for search, storage, and visualization. It even has container support!

Daniel Berman user avatar by
Daniel Berman
·
Oct. 26, 16 · Tutorial
Like (4)
Save
Tweet
Share
5.02K Views

Join the DZone community and get the full member experience.

Join For Free

sysdig is a powerful tool for linux system and container monitoring. capturing system activity directly from the kernel, sysdig allows process tracing and analysis and includes both a cli and ui for user interaction. as the folks at sysdig themselves put it: “think about sysdig as strace + tcpdump + htop + iftop + lsof + …awesome sauce.”

of course, the challenge is how to extract insights from this rich data once it is captured. we’re talking about thousands of messages logged per minute from the various processes that are running. how does one successfully monitor and analyze all of this information?

sysdig’s cloud solution includes various visualization and analysis features such as dashboards and alerts, but if you opt for the open source sysdig, you’re going to need to tap into an external data aggregation and analysis tool.

in comes the elk stack ( elasticsearch , logstash , and kibana ).

this post checks out the integration between sysdig and elk and details how to set up a local logging pipeline from sysdig to logstash to elasticsearch and kibana.

my setup: ubuntu 14.04, java sdk 1.7, elasticsearch 2.4, logstash 2.4, and kibana 4.5.

installing sysdig

there are a number of ways to install sysdig — either manually, automatically, or in a docker container. in this case, i used the following automatic installation script:

$ curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash


to verify sysdig was installed correctly, simply run it with:

$ sudo sysdig


you should be seeing a live tail of all the messages being traced by sysdig:

while sysdig does allow you to trace this output to a file, it is virtually impossible to access these log lines easily for analysis and visualization purposes.

configuring logstash

the next step is to configure logstash to pull the sysdig output and forward it to elasticsearch for indexing.

to do this, i’m first going to create a new ‘ sysdig-logstash.conf ’ file:

$ sudo vim /etc/logstash/sysdig-logstash.conf


then, i’m going to use the following logstash configuration:

input {
   stdin { }
}

filter {
      grok {
      pattern => "^%{number:num:int} %{number:time:float} %{int:cpu:int} %{notspace:procname} %{notspace:tid} (?<direction>[<>]) %{word:event} %{data:args}$"
   }

date {
   match => [ "time", "unix" ]
}

   if [args] {
      kv {
         source => "args"
         remove_field => "args"
      }
   }
}

output {
   #stdout { codec => rubydebug }
   elasticsearch {
      }
}


this configuration defines stdin as the input, applies a grok filter to the data, and sets a locally installed elasticsearch instance as the output. of course, if you’re using a remote deployment of elasticsearch, you’ll want to add a “hosts” configuration line in the output section.

after i save the file, i will then enter the following commands to run sysdig and logstash with the configuration file above (before you run this command, verify the path to both the binary and configuration files):

$ cd /opt/logstash

$ sysdig -t a "not(proc.name = sysdig)" | bin/logstash -f /etc/logstash/sysdig-logstash.conf


you should see logstash output with the following message:

settings: default pipeline workers: 1
logstash startup completed


analyzing and visualizing the data

our next step is to begin analyzing the data in kibana, elk’s user interface.

in a vanilla installation, and if this is the first time you are defining a logstash index, the settings tab will be displayed, in which the logstash-* index should already be identified by kibana:

configure kibana index pattern

after creating the new index, and opening the discover tab in kibana — you should be seeing all the data traced by sysdig and outputted to stdin:

sysdig outputs to stdin

to start to understand the potential of using sysdig with elk, let’s begin by adding some fields to the log display. i’m going to select the “ procname ” and “ event ” fields from the list of available fields on the left:

select event fields in log display

we can now see the different processes being traced and which event they triggered. i recommend reading up on how to interpret sysdig data before you begin analysis.

next up — visualizing the data. this is where the fun starts. kibana is renowned for its visualization features, but the truth is that it’s not always easy to fine-tune the visualizations to reflect exactly what we want to see in them (here’s a useful guide on how to create visualizations in kibana ).

number of messages

let’s start with something simple. i’m going to create a simple metric visualization, showing the number of messages being traced by sysdig in my machine.

to do this, i’m going to select the visualize tab, select the metric visualization type, use a new search, and then select count as the aggregation type:

count visualization type

top processes

another easy example is a pie chart that depicts the top processes that are taking place in one’s system. to create this visualization, select the pie chart visualization type and use the following configuration using the ‘ procname.raw ’ field:

top processes visualization

not surprisingly, with both elasticsearch and logstash running on the same system, java is ranking first at the top of the list.

and so on. once you have a series of these visualizations, collect them into one comprehensive dashboard for real-time system monitoring.

reservations

before i sign off, here are a few reasons why the title of this post contains the word “potentially.”

logstash parsing errors

after logging sysdig output into elk for a while, i noticed this parsing error being reported:

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"field name [.....y....5.k.../.......e,..d...............] cannot contain '.'"}


this is because some of the fields outputted by sysdig contain a dot ‘.’ a way around this is to install the logstash de_dot plugin , a process not described here.

logstash performance

the amount of data being shipped by sysdig is quite large, and logstash is prone to caving under heavy loads. in a sandbox type of environment, you can afford to ignore this. but if you scale the setup upwards a bit, you’re going to need some kind of buffering system.

analyzed fields

elasticsearch mapping configurations are recommended to enhance the analysis of sysdig data in kibana. out of the box, many sysdig fields such as “ procname ” are reported as analyzed, meaning that they contain analyzed strings. these, in turn, consume a lot of memory and also do not visualize as expected (please note that in the pie chart visualization above, i used the ‘ prodname_raw ‘ field instead of ‘ procname ‘).

summary

despite these reservations, there’s no doubt that with a bit of fine-tuning and the application of extra configurations, the combination between sysdig and elk offers great potential for a powerful solution that monitors systems and containers. with the soaring popularity of elk and the growing need for effective system and container monitoring, i’m sure a tighter integration between the two is forthcoming.

anyone picking up the gauntlet?

Visualization (graphics) Data (computing) Kibana Elasticsearch

Published at DZone with permission of Daniel Berman, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • The Importance of Delegation in Management Teams
  • The Future of Cloud Engineering Evolves
  • Top 12 Technical Skills Every Software Tester Must Have
  • Secrets Management

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: