Over a million developers have joined DZone.

Generate Common Data Reports With ELK

DZone's Guide to

Generate Common Data Reports With ELK

Want to get your hands on your database metrics? Here's a way to generate some quick and easy reports with ELK.

· Database Zone
Free Resource

Whether you work in SQL Server Management Studio or Visual Studio, Redgate tools integrate with your existing infrastructure, enabling you to align DevOps for your applications with DevOps for your SQL Server databases. Discover true Database DevOps, brought to you in partnership with Redgate.

Usually, critical data is stored in your databases. We can periodically poll and combine key metrics from the DB into a data report, such as how many tables/collections there are, how many items in each critical table, etc. Data report enables people to visualize the history and trend of key metrics.

Furthermore, if the basic data report can be generated automatically, people can easily add more metrics, which may make more business sense. The most exciting part is that we can get a general solution which applies to different projects! So how we can get this idea started?


Step 1: Define Key Metrics to Collect and Display

Here are metrics I've been actively using.

  1. TOTAL_ITEM_COUNT: The sum of the item count in all critical tables/collections. This gives us a rough understanding of how much data we have in general.
  2. TABLE_ITEM_COUNT: Item count for each major table/collection and its history. It shows how fast the data grows from day to day.
  3. TOTAL_DISK_OCCUPIED: Total occupied disk capacity. The physical measurement of our data helps us to do a better capacity planning.

All of the above metrics give project owners a general estimation for current status and some sort of prediction.

People might have problems defining suitable metrics to fit their projects in the beginning. Don't worry. Once we've finished the basic version, we can revisit it and add fancier metrics later.

Step 2: Collect Metrics from the DB

The management tools (CLI/API) of different databases may be different. The way how we calculate each metric may be different as well. The good thing is that we can always unify different metrics into the format below:

[Date, ItemName, PropName, PropValue]

# Sample:
# 2016-07-11 14:15:20 my_table1 MYSQLItemNum 2000
# 2016-07-11 15:15:18 es_index1 ESItemNum 500

Step 3: Feed Data to ELK and Draw Useful Diagrams


To combine all the pieces together, here is a sample report. In the below example, the system uses CouchBase and Elasticsearch as Key DB stores.


It’s easier than you think to extend DevOps practices to SQL Server with Redgate tools. Discover how to introduce true Database DevOps, brought to you in partnership with Redgate

devops ,metrics ,report ,elk

Published at DZone with permission of Denny Zhang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.


Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.


{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}