Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Generate Common Data Reports With ELK

DZone's Guide to

Generate Common Data Reports With ELK

Want to get your hands on your database metrics? Here's a way to generate some quick and easy reports with ELK.

· Database Zone
Free Resource

Learn how our document data model can map directly to how you program your app, and native database features like secondary indexes, geospatial and text search give you full access to your data. Brought to you in partnership with MongoDB.

Usually, critical data is stored in your databases. We can periodically poll and combine key metrics from the DB into a data report, such as how many tables/collections there are, how many items in each critical table, etc. Data report enables people to visualize the history and trend of key metrics.

Furthermore, if the basic data report can be generated automatically, people can easily add more metrics, which may make more business sense. The most exciting part is that we can get a general solution which applies to different projects! So how we can get this idea started?

db_report.png


Step 1: Define Key Metrics to Collect and Display

Here are metrics I've been actively using.

  1. TOTAL_ITEM_COUNT: The sum of the item count in all critical tables/collections. This gives us a rough understanding of how much data we have in general.
  2. TABLE_ITEM_COUNT: Item count for each major table/collection and its history. It shows how fast the data grows from day to day.
  3. TOTAL_DISK_OCCUPIED: Total occupied disk capacity. The physical measurement of our data helps us to do a better capacity planning.

All of the above metrics give project owners a general estimation for current status and some sort of prediction.

People might have problems defining suitable metrics to fit their projects in the beginning. Don't worry. Once we've finished the basic version, we can revisit it and add fancier metrics later.

Step 2: Collect Metrics from the DB

The management tools (CLI/API) of different databases may be different. The way how we calculate each metric may be different as well. The good thing is that we can always unify different metrics into the format below:

[Date, ItemName, PropName, PropValue]

# Sample:
# 2016-07-11 14:15:20 my_table1 MYSQLItemNum 2000
# 2016-07-11 15:15:18 es_index1 ESItemNum 500


Step 3: Feed Data to ELK and Draw Useful Diagrams

data_report_design.png


To combine all the pieces together, here is a sample report. In the below example, the system uses CouchBase and Elasticsearch as Key DB stores.

db_report_sample.png

Discover when your data grows or your application performance demands increase, MongoDB Atlas allows you to scale out your deployment with an automated sharding process that ensures zero application downtime. Brought to you in partnership with MongoDB.

Topics:
devops ,metrics ,report ,elk

Published at DZone with permission of Denny Zhang, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

THE DZONE NEWSLETTER

Dev Resources & Solutions Straight to Your Inbox

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

X

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}