Sampler: Dashboards, Monitoring, Alerting From Your Terminal

As a backend developer, I always need to monitor something — the state machine in the database, records count, message queue lag, custom application metrics, system performance, progress of my deployment scripts.

For a long time, I was trying to find a tool that could do keep everything in one place, alert me on a trigger, and be as fast and nimble as a local development tool. Heavy production monitoring systems can't help with such tasks; I needed a swiss army knife — something that could be configured in a minute and give me results right away.

I couldn't find anything, so I wrote my own. Meet Sampler — a tool for shell commands execution, visualization, and alerting configured with a simple YAML file.

To install Sampler, just follow the instructions for macOS, Linux or Windows

How Does it Work?

The idea is pretty simple — almost all metrics you might want to monitor and visualize are available through the CLI:

• The obvious ones: CPU, memory, disk space, network.
• Telemetry from a remote machine via ssh.
• Any database metrics and results of your custom queries.
• JMX.
• Everything available via http, e.g. Spring Boot Actuator metrics.
• Kafka, RabbitMQ, ActiveMQ.
• Docker.

That means we can create our dashboards without complex integrations, drivers, and custom code; we can do it right from the terminal.

Using Sampler is basically a 3-step process:

• Define your configuration in a YAML file.
• Run sampler -c your-config-file.yml.
• Adjust components size and location on UI.

Sampler executes specified commands with a required rate. The output is used for visualization.

Examples

Visualizations and Their YAML Configuration

runcharts:
  - title: Search engine response time
    rate-ms: 500        # sampling rate, default = 1000
    scale: 2            # number of digits after sample decimal point, default = 1
    legend:
      enabled: true     # enables item labels, default = true
      details: false    # enables item statistics: cur/min/max/dlt values, default = true
    items:
      - label: GOOGLE
        sample: curl -o /dev/null -s -w '%{time_total}'  https://www.google.com
        color: 178      # 8-bit color number, default one is chosen from a pre-defined palette
      - label: YAHOO
        sample: curl -o /dev/null -s -w '%{time_total}'  https://search.yahoo.com
      - label: BING
        sample: curl -o /dev/null -s -w '%{time_total}'  https://www.bing.com

Sparkline

sparklines:
  - title: CPU usage
    rate-ms: 200
    scale: 0
    sample: ps -A -o %cpu | awk '{s+=$1} END {print s}'
  - title: Free memory pages
    rate-ms: 200
    scale: 0
    sample: memory_pressure | grep 'Pages free' | awk '{print $3}'

Barchart

barcharts:
  - title: Local network activity
    rate-ms: 500        # sampling rate, default = 1000
    scale: 0            # number of digits after sample decimal point, default = 1
    items:
      - label: UDP bytes in
        sample: nettop -J bytes_in -l 1 -m udp | awk '{sum += $4} END {print sum}'
      - label: UDP bytes out
        sample: nettop -J bytes_out -l 1 -m udp | awk '{sum += $4} END {print sum}'
      - label: TCP bytes in
        sample: nettop -J bytes_in -l 1 -m tcp | awk '{sum += $4} END {print sum}'
      - label: TCP bytes out
        sample: nettop -J bytes_out -l 1 -m tcp | awk '{sum += $4} END {print sum}'

Gauge

gauges:
  - title: Minute progress
    rate-ms: 500        # sampling rate, default = 1000
    scale: 2            # number of digits after sample decimal point, default = 1
    percent-only: false # toggle display of the current value, default = false
    color: 178          # 8-bit color number, default one is chosen from a pre-defined palette
    cur:
      sample: date +%S  # sample script for current value
    max:
      sample: echo 60   # sample script for max value
    min:
      sample: echo 0    # sample script for min value
  - title: Year progress
    cur:
      sample: date +%j
    max:
      sample: echo 365
    min:
      sample: echo 0

Textbox

textboxes:
  - title: Local weather
    rate-ms: 10000      # sampling rate, default = 1000
    sample: curl wttr.in?0ATQF
    border: false       # border around the item, default = true
    color: 178          # 8-bit color number, default is white
  - title: Docker containers stats
    rate-ms: 500
    sample: docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.PIDs}}"

Asciibox

asciiboxes:
  - title: UTC time
    rate-ms: 500        # sampling rate, default = 1000
    font: 3d            # font type, default = 2d
    border: false       # border around the item, default = true    
    color: 43           # 8-bit color number, default is white
    sample: env TZ=UTC date +%r

Triggers

Sampler can execute conditional actions based on your shell commands output. Using triggers, you can start the deployment process and get a notification when it's done. You can also get an alert when your metric exceeds a threshold.

Sampler supports four types of trigger actions:

• Visual (alert will be displayed on top of your chart).
• Sound (NASA quindar tone).
• Standard terminal bell.
• Custom script, where you can use an alert label for current and previous values.

Here is an example of a chart with a configured search engine latency alert. When the latency exceeds the specified threashold of 0.3 sec, Sampler will ring a bell, show the visual notification, and execute a custom script, which uses macOS voice capabilities to say the latency value:

runcharts:
  - title: SEARCH ENGINE RESPONSE TIME (sec)
    rate-ms: 200
    items:
      - label: GOOGLE
        sample: curl -o /dev/null -s -w '%{time_total}'  https://www.google.com
      - label: YAHOO
        sample: curl -o /dev/null -s -w '%{time_total}'  https://search.yahoo.com     
    triggers:
      - title: Latency threshold exceeded
        condition: echo "$prev < 0.3 && $cur > 0.3" |bc -l  # expects "1" as TRUE indicator
        actions:
          terminal-bell: true  # standard terminal bell, default = false
          sound: true   # NASA quindar tone, default = false
          visual: true  # visual notification on top of the component area, default = false
          script: 'say alert: ${label} latency exceeded ${cur} second' # an arbitrary script, which can use $cur, $prev and $label variables

There's More

Interactive shell support, PTY mode, variables, and real-world recipes to work with MySQL, PostgreSQL, MongoDB, Neo4J, Kafka, etc. You can read all about Sampler's advanced capabilities in the documentation.

Don't hesitate to ask questions, contribute, and star the project on Github