DZone
Big Data Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Big Data Zone > Storing SysDig Streams with Apache NiFi

Storing SysDig Streams with Apache NiFi

This is a tutorial on using Apache NiFi 1.0.0 to ingest, parse and store Linux system diagnostics and storing them in HDFS and file systems.

Tim Spann user avatar by
Tim Spann
CORE ·
Oct. 29, 16 · Big Data Zone · Tutorial
Like (3)
Save
Tweet
4.40K Views

Join the DZone community and get the full member experience.

Join For Free

SysDig (GitHub) is an open source tool that allows for the exploration, analysis and troubleshooting of Linux systems and containers. It is well documented and very easy to install and use. It can be used for container and Linux system diagnostics, security analysis, monitoring and basic system information capture. Remember that SysDig can produce thousands of lines of messages and can continue doing so infinitely depending on the options.   

To install:To get started, follow the get started guide and try out the csysdig console.

 curl -s https://s3.amazonaws.com/download.draios.com/stable/install-sysdig | sudo bash 

There are a number of examples of using SysDig for getting specific system information, this could include processes, networking, files, errors, events and more.  

It is a very compact tool that can generate a ton of system and tracing information to use for current or predictive analysis of your system.   My use case was to capture a stream of this data every 15 minutes and store to Apache Phoenix on HBase for predictive analytics.

Image title

To use from the command line:

sysdig --help
sysdig version 0.11.0
Usage: sysdig [options] [-p <output_format>] [filter]

To start my flow in NiFi 1.00, I use the ExecuteProcess processor and wrap my SysDig command in a BASH script. This example returns ASCII text formatted as JSON and just gives me a 1-second snapshot.  This actually generates a ton of data.

 sysdig -A -j -M 1 --unbuffered 

This is an example of an Event returned by SysDig in JSON format:

"evt.cpu":6,
"evt.dir":">",
"evt.info":"fd=7(<f>/usr/lib64/python2.7/lib-dynload/_elementtree.so) ",
"evt.num":111138,
"evt.outputtime":1477313882635597873,
"evt.type":"fstat",
"proc.name":"python",
"thread.tid":14602}

Resources:

  • Fishing for Hackers With Sysdig Part 1
  • Fishing for Hackers With Sysdig Part 2
  • Sysdig Examples
  • Sysdig Cheatsheet mapping to legacy tools
Apache NiFi Stream (computing)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • How BDD Works Well With EDA
  • Terraform Controller: Cloud Resource Self-Service
  • Model Quantization for Edge AI
  • Top 5 Programming Languages in AI: A Comparison

Comments

Big Data Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo