Over a million developers have joined DZone.

Generate Common Data Reports With ELK

DZone's Guide to

Generate Common Data Reports With ELK

Want to get your hands on your database metrics? Here's a way to generate some quick and easy reports with ELK.

· Database Zone
Free Resource

Read why times series is the fastest growing database category.

Usually, critical data is stored in your databases. We can periodically poll and combine key metrics from the DB into a data report, such as how many tables/collections there are, how many items in each critical table, etc. Data report enables people to visualize the history and trend of key metrics.

Furthermore, if the basic data report can be generated automatically, people can easily add more metrics, which may make more business sense. The most exciting part is that we can get a general solution which applies to different projects! So how we can get this idea started?


Step 1: Define Key Metrics to Collect and Display

Here are metrics I've been actively using.

  1. TOTAL_ITEM_COUNT: The sum of the item count in all critical tables/collections. This gives us a rough understanding of how much data we have in general.
  2. TABLE_ITEM_COUNT: Item count for each major table/collection and its history. It shows how fast the data grows from day to day.
  3. TOTAL_DISK_OCCUPIED: Total occupied disk capacity. The physical measurement of our data helps us to do a better capacity planning.

All of the above metrics give project owners a general estimation for current status and some sort of prediction.

People might have problems defining suitable metrics to fit their projects in the beginning. Don't worry. Once we've finished the basic version, we can revisit it and add fancier metrics later.

Step 2: Collect Metrics from the DB

The management tools (CLI/API) of different databases may be different. The way how we calculate each metric may be different as well. The good thing is that we can always unify different metrics into the format below:

[Date, ItemName, PropName, PropValue]

# Sample:
# 2016-07-11 14:15:20 my_table1 MYSQLItemNum 2000
# 2016-07-11 15:15:18 es_index1 ESItemNum 500

Step 3: Feed Data to ELK and Draw Useful Diagrams


To combine all the pieces together, here is a sample report. In the below example, the system uses CouchBase and Elasticsearch as Key DB stores.


Learn how to get 20x more performance than Elastic by moving to a Time Series database.

devops ,metrics ,report ,elk

Published at DZone with permission of Denny Zhang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}