Over a million developers have joined DZone.

Generate Common Data Reports With ELK

Want to get your hands on your database metrics? Here's a way to generate some quick and easy reports with ELK.

· Database Zone

Learn NoSQL for free with hands-on sample code, example queries, tutorials, and more.  Brought to you in partnership with Couchbase.

Usually, critical data is stored in your databases. We can periodically poll and combine key metrics from the DB into a data report, such as how many tables/collections there are, how many items in each critical table, etc. Data report enables people to visualize the history and trend of key metrics.

Furthermore, if the basic data report can be generated automatically, people can easily add more metrics, which may make more business sense. The most exciting part is that we can get a general solution which applies to different projects! So how we can get this idea started?

db_report.png


Step 1: Define Key Metrics to Collect and Display

Here are metrics I've been actively using.

  1. TOTAL_ITEM_COUNT: The sum of the item count in all critical tables/collections. This gives us a rough understanding of how much data we have in general.
  2. TABLE_ITEM_COUNT: Item count for each major table/collection and its history. It shows how fast the data grows from day to day.
  3. TOTAL_DISK_OCCUPIED: Total occupied disk capacity. The physical measurement of our data helps us to do a better capacity planning.

All of the above metrics give project owners a general estimation for current status and some sort of prediction.

People might have problems defining suitable metrics to fit their projects in the beginning. Don't worry. Once we've finished the basic version, we can revisit it and add fancier metrics later.

Step 2: Collect Metrics from the DB

The management tools (CLI/API) of different databases may be different. The way how we calculate each metric may be different as well. The good thing is that we can always unify different metrics into the format below:

[Date, ItemName, PropName, PropValue]

# Sample:
# 2016-07-11 14:15:20 my_table1 MYSQLItemNum 2000
# 2016-07-11 15:15:18 es_index1 ESItemNum 500


Step 3: Feed Data to ELK and Draw Useful Diagrams

data_report_design.png


To combine all the pieces together, here is a sample report. In the below example, the system uses CouchBase and Elasticsearch as Key DB stores.

db_report_sample.png

The Getting Started with NoSQL Guide will get you hands-on with NoSQL in minutes with no coding needed. Brought to you in partnership with Couchbase.

Topics:
devops ,metrics ,report ,elk

Published at DZone with permission of Denny Zhang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

SEE AN EXAMPLE
Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.
Subscribe

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}