{{announcement.body}}
{{announcement.title}}

Sysdig: What It Is and How to Use It

DZone 's Guide to

Sysdig: What It Is and How to Use It

This tutorial goes over the fundamentals of using Sysdig to get a clear understanding of the activity generated by your containers.

· Cloud Zone ·
Free Resource

Sysdig is a universal system visibility tool with support for containers. What makes Sysdig special, is that it hooks itself into the machine's kernel and segregates the information on a per-container basis.
For the scope of this tutorial, we will focus on the open-source version of Sysdig.

In the next sections, you will:

  • Install Sysdig
  • Spin up a Wordpress installation using docker-compose
  • Use Sysdig to collect events and analyze them at a later time
  • Use Sysdig to analyze data in real-time

Prerequisites

  • Docker is installed on your system. For details about installing Docker, refer to the Install Docker page.
  • Docker Compose is installed on your system. Refer to the Install Docker Compose page for instructions about how to install Docker Compose.
  • The kernel headers are installed on the host system.

Install Sysdig

Follow these steps to install Sysdig inside a Docker container:

  1. In a terminal window, execute the following command to pull the Sysdig Docker image:
Shell
 




xxxxxxxxxx
1


1
docker pull sysdig/sysdig


Plain Text
 




xxxxxxxxxx
1
16


 
1
Using default tag: latest 
2
latest: Pulling from sysdig/sysdig 
3
2967486b0658: Pull complete 
4
78101b780c72: Pull complete 
5
7e78b657334d: Pull complete 
6
650327159ca8: Pull complete 
7
47ebf73ab754: Pull complete 
8
bf51ac76a6d9: Pull complete 
9
0cd11104dbf6: Pull complete 
10
e6dcf17d00d8: Pull complete 
11
230d60083576: Pull complete 
12
fd5ea9faf384: Pull complete 
13
6de86c8ed6e9: Pull complete 
14
8d1825f8be4b: Pull complete 
15
Digest: sha256:bbfe6953fd2b3221a8974eb13024dd33c7e78aebef8fee3d7a0d9ecdeed84ce0 
16
Status: Downloaded newer image for sysdig/sysdig:latest


  1. Run Sysdig in a container by entering:
Shell
 




xxxxxxxxxx
1


 
1
docker run -i -t --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig


Plain Text
 




xxxxxxxxxx
1
13


 
1
* Setting up /usr/src links from host 
2
* Unloading sysdig-probe, if present 
3
* Running dkms install for sysdig 
4
Error! echo 
5
Your kernel headers for kernel 3.10.0-957.12.2.el7.x86_64 cannot be found at 
6
/lib/modules/3.10.0-957.12.2.el7.x86_64/build or /lib/modules/3.10.0-957.12.2.el7.x86_64/source. 
7
* Running dkms build failed, couldn't find /var/lib/dkms/sysdig/0.26.4/build/make.log 
8
* Trying to load a system sysdig-probe, if present 
9
* Trying to find precompiled sysdig-probe for 3.10.0-957.12.2.el7.x86_64 
10
Found kernel config at /host/boot/config-3.10.0-957.12.2.el7.x86_64 
11
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.26.4-x86_64-3.10.0-957.12.2.el7.x86_64-82e2ae1fb159132636f7b50a762f20ef.ko 
12
Download succeeded, loading module 
13
root@7b14a23f22eb:/#


A few things to note about the above command:

  • The -i flag keeps STDIN open.
  • The --privileged parameter provides access to all devices on the host. Also it sets SELinux to allow the processes running inside of the container the same access to the host as a process running on the host.
  • The -v flag specifies the list of files (on the host) that Sysdig can access.

Spin Up a Wordpress Installation

In this section, you will install Wordpress using the docker-compose command.

  1. In a new terminal window, move into your projects directory and type the following commands:
Shell
 




xxxxxxxxxx
1


1
mkdir wordpress-sysdig && cd wordpress-sysdig


Create a file called docker-compose with the following content:
YAML
 




xxxxxxxxxx
1
28


 
1
version: '3.3' 
2
 
             
3
services:   
4
  db:     
5
    image: mysql:5.7     
6
    volumes:       
7
      - db_data:/var/lib/mysql     
8
    restart: always     
9
    environment:       
10
      MYSQL_ROOT_PASSWORD: somewordpress       
11
      MYSQL_DATABASE: wordpress       
12
      MYSQL_USER: wordpress       
13
      MYSQL_PASSWORD: wordpress    
14
 
             
15
wordpress:     
16
  depends_on:       
17
    - db     
18
  image: wordpress:latest     
19
  ports:       
20
    - "8000:80"     
21
  restart: always     
22
  environment:       
23
    WORDPRESS_DB_HOST: db:3306       
24
    WORDPRESS_DB_USER: wordpress       
25
    WORDPRESS_DB_PASSWORD: wordpress       
26
    WORDPRESS_DB_NAME: wordpress 
27
volumes:    
28
  db_data: {}


  1. Run the docker-compose up command in detached mode with:
Shell
 




xxxxxxxxxx
1


 
1
docker-compose up -d


Plain Text
 




xxxxxxxxxx
1
29


 
1
Creating network "wordpress-sysdig_default" with the default driver
2
Creating volume "wordpress-sysdig_db_data" with default driver
3
Pulling wordpress (wordpress:latest)...
4
latest: Pulling from library/wordpress
5
8ec398bc0356: Pull complete 
6
85cf4fc86478: Pull complete 
7
970dadf4ccb6: Pull complete 
8
8c04561117a4: Pull complete 
9
d6b7434b63a2: Pull complete 
10
83d8859e9744: Pull complete 
11
9c3d824d0ad5: Pull complete 
12
9e316fd5b3b3: Pull complete 
13
578b40496c37: Pull complete 
14
814ae7711d3c: Pull complete 
15
4896fed78b6b: Pull complete 
16
e74d71e9611d: Pull complete 
17
46017765526c: Pull complete 
18
280386098458: Pull complete 
19
f32eb0d8c540: Pull complete 
20
5c47b9ea747a: Pull complete 
21
ecda5b7aad12: Pull complete 
22
84256a6b6b44: Pull complete 
23
35d4f385efb7: Pull complete 
24
bf697c2ae701: Pull complete 
25
d054b015f084: Pull complete 
26
Digest: sha256:73e8d8adf491c7a358ff94c74c8ebe35cb5f8857e249eb8ce6062b8576a01465 
27
Status: Downloaded newer image for wordpress:latest 
28
Creating wordpress-sysdig_db_1 ... done 
29
Creating wordpress-sysdig_wordpress_1 ... done


  1. You can verify the status of your containers with:
Shell
 




xxxxxxxxxx
1


1
docker ps


If all is going well, you should see something similar to the following output:

Shell
 




xxxxxxxxxx
1


1
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS              PORTS                  NAMES
2
f390eec29f52        wordpress:latest    "docker-entrypoint.s…"   About a minute ago   Up About a minute   0.0.0.0:8000->80/tcp   wordpress-sysdig_wordpress_1
3
a844840626d8        mysql:5.7           "docker-entrypoint.s…"   About a minute ago   Up About a minute   3306/tcp, 33060/tcp    wordpress-sysdig_db_1
4
7b14a23f22eb        sysdig/sysdig       "/docker-entrypoint.…"   13 minutes ago       Up 13 minutes                              sysdig


  1. Now Wordpress is up and running. Point your browser to http://localhost:8000 to start the installation wizard:

  1. Once the installation wizard is finished, let us go ahead and create a sample post:

Collecting Data to a File

In this section, we'll show how you can use Sysdig to collect events and analyze them at a later time.

  1. To dump all captured events to a file, move to the Sysdig container, and enter the following command:
sysdig -w monitoring-wordpress.scap
  1. In a new terminal window, use ab to make 10000 requests with a maximum of 100 requests running concurrently:
ab -n 1000 -c 100 http://localhost:8000/?p=7
This is ApacheBench, Version 2.3 <$Revision: 1430300 $> 
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ 
Licensed to The Apache Software Foundation, http://www.apache.org/ 

Benchmarking localhost (be patient) 
Completed 100 requests 
Completed 200 requests 
Completed 300 requests 
Completed 400 requests 
Completed 500 requests 
Completed 600 requests 
Completed 700 requests 
Completed 800 requests 
Completed 900 requests 
Completed 1000 requests 
Finished 1000 requests

Note that the above output was truncated for brevity.

  1. Move back to tour Sysdig container and stop capturing data by entering "CTRL+C".

Analyzing data

Now, if you look at the size of the monitoring-wordpress.scap file, you'll notice that Sysdig captured no less than 80M of data:

Shell
 




xxxxxxxxxx
1


1
ls -lh monitoring-wordpress.scap


Plain Text
 




xxxxxxxxxx
1


1
-rw-r--r--. 1 root root 80M Jan  7 16:28 monitoring-wordpress.scap


To find your way through this mountain of data, you'll use something called a chisel.

A chisel is basically a Lua script that analyzes the event stream and performs useful actions.

You can run the following command to display the list of chisels:

Shell
 




xxxxxxxxxx
1


 
1
sysdig -cl


Shell
 




xxxxxxxxxx
1
20


 
1
Category: Application 
2
--------------------- 
3
httplog         HTTP requests log httptop         
4
Top HTTP requests memcachelog     
5
memcached requests log 
6
 
             
7
Category: CPU Usage 
8
------------------- 
9
spectrogram     Visualize OS latency in real time. 
10
subsecoffset    Visualize subsecond offset execution time. 
11
topcontainers_cpu                
12
      Top containers by CPU usage topprocs_cpu    
13
      Top processes by CPU usage 
14
 
             
15
Category: Errors 
16
---------------- 
17
topcontainers_error                
18
      Top containers by number of errors 
19
topfiles_errors Top files by number of errors 
20
topprocs_errors top processes by number of errors 


Note that the above output was truncated for brevity.

To retrieve detailed information about a chisel, run the sysdig command followed by the -i flag and the name of the chisel, as in the following example:

Shell
 




xxxxxxxxxx
1


1
sysdig -i httptop


Plain Text
 




xxxxxxxxxx
1


 
1
Category: Application 
2
--------------------- 
3
httptop         Top HTTP requests 
4
 
             
5
Show top HTTP requests by: ncalls, time or bytes 
6
Args: 
7
[string] by - Show top HTTP transactions by: ncalls, time or by                
8
        tes, default is ncalls


Continuing our example, here's how you can use the httptop chisel to display the top HTTP requests:

Shell
 




xxxxxxxxxx
1


1
sysdig -r monitoring-wordpress.scap -c httptop


Plain Text
 




xxxxxxxxxx
1
11


 
1
ncalls              method              url 
2
-------------------------------------------------------------------------------- 
3
2001                GET                 localhost:8000/?p=7 
4
14                  OPTIONS             * 
5
2                   GET                 localhost:8000/favicon.ico 
6
1                   GET                 /wp-content/themes/twentytwenty/assets/fonts/inter/Inter-upright-var.woff2 
7
1                   GET                 localhost/v1.24/containers/6bd8418eb03f/json 
8
1                   GET                 localhost/v1.24/containers/06def7875617/json 
9
1                   GET                 /v1.24/images/1b1624b63467ec61fab209b6be6e79707ae786df86607b9474b246acd31600 
10
1                   GET                 /v1.24/images/db39680b63ac47a1d989da7b742f7b382af34d85a68214f8977bad59c05901 
11
1                   GET                 localhost:8000/


You can see the same information in a container-friendly format with the -pcontainer flag:

Shell
 




xxxxxxxxxx
1


 
1
sysdig -r monitoring-wordpress.scap -c httptop -pcontainer


Plain Text
 




xxxxxxxxxx
1
10


 
1
ncalls              container           method              url 
2
-------------------------------------------------------------------------------- 
3
1000                wordpress-sysdig_wo GET                 localhost:8000/?p=7 
4
1000                host                GET                 localhost:8000/?p=7 
5
43                  wordpress-sysdig_wo OPTIONS             * 
6
1                   sysdig              GET                 /v1.24/images/1b1624b63467ec61fab209b6be6e79707ae786df86607b9474b246acd31600 
7
1                   sysdig              GET                 localhost/v1.24/containers/06def7875617/json 
8
1                   sysdig              GET                 localhost/v1.24/containers/cd06093b141b/json 
9
1                   sysdig              GET                 /v1.24/images/00e230fe24da9067f9b6e65cfbe9935a5affac1ae8e44edb6a5b0ccc26374d 
10
1                   sysdig              GET                 /v1.24/images/db39680b63ac47a1d989da7b742f7b382af34d85a68214f8977bad59c05901


Digging Deeper

Sysdig captures content-rich information that lets you get detailed insights into the inner-workings of your containers. Let's suppose you're running a few containers and want to know which process consumes the most CPU.

  1. List the containers that were active during the period in which you captured events:
Shell
 




xxxxxxxxxx
1


1
sysdig -r monitoring-wordpress.scap -c lscontainers


You can identify the container that consumed the most CPU with:
Shell
 




xxxxxxxxxx
1


1
sysdig -r monitoring-wordpress.scap -c topcontainers_cpu


Plain Text
 




xxxxxxxxxx
1


 
1
CPU%                container.name 
2
-------------------------------------------------------------------------------- 
3
5.37%               wordpress-sysdig_wordpress_1 
4
1.35%               wordpress-sysdig_db_1 
5
0.84%               host 
6
0.51%               sysdig


  1. You can dig even deeper and identify the most CPU intensive process with the topprocs_cpu chisel:
Shell
 




xxxxxxxxxx
1


 
1
sysdig -r monitoring-wordpress.scap -c topprocs_cpu container.name contains wordpress_1


Plain Text
 




xxxxxxxxxx
1
12


 
1
CPU%                Process             PID 
2
-------------------------------------------------------------------------------- 
3
0.12%               apache2             8383 
4
0.11%               apache2             9413 
5
0.11%               apache2             9300 
6
0.11%               apache2             9242 
7
0.11%               apache2             8897 
8
0.11%               apache2             8422 
9
0.10%               apache2             9372 
10
0.10%               apache2             9241 
11
0.10%               apache2             8424 
12
0.09%               apache2             9429


If you want to see more details,  the ps chisel provides a more verbose alternative:

Shell
 




xxxxxxxxxx
1


 
1
sysdig -r monitoring-wordpress.scap -c ps container.name=wordpress-sysdig_wordpress_1


Plain Text
 




xxxxxxxxxx
1
12


 
1
TID     PID     USER        VIRT       RES        FDLIMIT   CMD 
2
5896    5896    root        232.82M    22.32M     429496729 apache2 
3
8383    8383    www-data    307.44M    25.46M     429496729 apache2 
4
8422    8422    www-data    235.44M    22.90M     429496729 apache2 
5
8424    8424    www-data    307.44M    25.46M     429496729 apache2 
6
8897    8897    www-data    235.44M    22.89M     429496729 apache2 
7
9154    9154    www-data    235.44M    22.91M     429496729 apache2 
8
9241    9241    www-data    307.44M    25.66M     429496729 apache2 
9
9242    9242    www-data    307.44M    25.67M     429496729 apache2 
10
9300    9300    www-data    235.44M    22.89M     429496729 apache2 
11
9372    9372    www-data    235.44M    22.89M     429496729 apache2 
12
9413    9413    www-data    233.44M    20.77M     429496729 apache2


Useful Tips

If you run Sysdig to capture events as in the above example (sysdig -w monitoring-wordpress.scap), the event file will grow continuously until it consumes all the available space. There are a few methods that can help prevent this from happening:

  • Specify the number of events Sysdig should capture by passing it the -n flag. Once Sysdig captures the specified number of events, it'll automatically exit:
Shell
 




xxxxxxxxxx
1


1
sysdig -n 5000 -w monitoring-wordpress.scap


Use the -C flag to configure Sysdig so that it breaks the capture into smaller files of a specified size. The following example continuously saves events to files < 10MB:
Shell
 




xxxxxxxxxx
1


1
sysdig -C 10 -w monitoring-wordpress.scap


This will create a bunch of files no larger than 10 MB:

Shell
 




xxxxxxxxxx
1


 
1
ls -lh monitoring-wordpress*


Plain Text
 




xxxxxxxxxx
1


 
1
-rw-r--r--. 1 root root 9.6M Jan  7 17:13 monitoring-wordpress.scap0 
2
-rw-r--r--. 1 root root 9.6M Jan  7 17:14 monitoring-wordpress.scap1 
3
-rw-r--r--. 1 root root 9.6M Jan  7 17:14 monitoring-wordpress.scap2 
4
-rw-r--r--. 1 root root 9.6M Jan  7 17:14 monitoring-wordpress.scap3 
5
-rw-r--r--. 1 root root 9.6M Jan  7 17:14 monitoring-wordpress.scap4 
6
-rw-r--r--. 1 root root 9.6M Jan  7 17:14 monitoring-wordpress.scap5 
7
-rw-r--r--. 1 root root 9.6M Jan  7 17:14 monitoring-wordpress.scap6 
8
-rw-r--r--. 1 root root 9.6M Jan  7 17:14 monitoring-wordpress.scap7 
9
-rw-r--r--. 1 root root 6.4M Jan  7 17:14 monitoring-wordpress.scap8


  • Specify the maximum number of files Sysdig should keep with the -W flag. For example, you can combine the -C and -W flags like so:
Shell
 




xxxxxxxxxx
1


1
sysdig -C 10 -W 4 -w monitoring-wordpress.scap


The above command will only keep the last four capture files:

Shell
 




xxxxxxxxxx
1


1
ls -lh monitoring-wordpress*


Plain Text
 




xxxxxxxxxx
1


 
1
-rw-r--r--. 1 root root 7.2M Jan  7 17:21 monitoring-wordpress.scap0 
2
-rw-r--r--. 1 root root 9.6M Jan  7 17:21 monitoring-wordpress.scap1 
3
-rw-r--r--. 1 root root 9.6M Jan  7 17:21 monitoring-wordpress.scap2 
4
-rw-r--r--. 1 root root 9.6M Jan  7 17:21 monitoring-wordpress.scap3 
5
root@cd06093b141b:/# sysdig -C 10 -W 4 -w monitoring-wordpress.scap
6
 
             


Real-Time Monitoring

With Sysdig, you can also analyze data real-time. At first glance, this can seem like a daunting task because, by default, all events are continuously printed out to the console. Fortunately, chisels are here to help.

Let's take an example.

Analyze Processes on a Per Container Basis

  1. Run the following command to list your containers:
Shell
 




xxxxxxxxxx
1


 
1
docker ps


Plain Text
 




xxxxxxxxxx
1


 
1
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                  NAMES 
2
5b253e74e8e7        sysdig/sysdig       "/docker-entrypoint.…"   9 minutes ago       Up 9 minutes                               sysdig 
3
 
             
4
06def7875617        wordpress:latest    "docker-entrypoint.s…"   3 hours ago         Up 3 hours          0.0.0.0:8000->80/tcp   wordpress-sysdig_wordpress_1 
5
 
             
6
6bd8418eb03f        mysql:5.7           "docker-entrypoint.s…"   3 hours ago         Up 3 hours          3306/tcp, 33060/tcp    wordpress-sysdig_db_1


  1. You can analyze the processes running in the WordPress container with:
Shell
 




xxxxxxxxxx
1


1
sysdig -pc -c topprocs_cpu container.name=wordpress-sysdig_wordpress_1



  1. Similarly, you can analyze the processes running in the MySQL container:
Shell
 




xxxxxxxxxx
1


1
sysdig -pc -c topprocs_cpu container.name=wordpress-sysdig_db_1



Note that, not much different from this example, Sysdig can monitor network traffic, disk usage, and so on.


In this tutorial, you have gone over the fundamentals of using Sysdig to get a clear understanding of the activity generated by your containers. The examples in this blog post helped you get started and, in future tutorials, we'll show you how to use Csysdig and Sysdig Inspect.

Topics:
cloud (add topic), docker, kubernetes, linux, programming, serverless, sysdig

Published at DZone with permission of Sudip Sengupta . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}