Usually, critical data is stored in your databases. We can periodically poll and combine key metrics from the DB into a data report, such as how many tables/collections there are, how many items in each critical table, etc. Data report enables people to visualize the history and trend of key metrics.
Furthermore, if the basic data report can be generated automatically, people can easily add more metrics, which may make more business sense. The most exciting part is that we can get a general solution which applies to different projects! So how we can get this idea started?
Step 1: Define Key Metrics to Collect and Display
Here are metrics I've been actively using.
- TOTAL_ITEM_COUNT: The sum of the item count in all critical tables/collections. This gives us a rough understanding of how much data we have in general.
- TABLE_ITEM_COUNT: Item count for each major table/collection and its history. It shows how fast the data grows from day to day.
- TOTAL_DISK_OCCUPIED: Total occupied disk capacity. The physical measurement of our data helps us to do a better capacity planning.
All of the above metrics give project owners a general estimation for current status and some sort of prediction.
People might have problems defining suitable metrics to fit their projects in the beginning. Don't worry. Once we've finished the basic version, we can revisit it and add fancier metrics later.
Step 2: Collect Metrics from the DB
The management tools (CLI/API) of different databases may be different. The way how we calculate each metric may be different as well. The good thing is that we can always unify different metrics into the format below:
[Date, ItemName, PropName, PropValue] # Sample: # 2016-07-11 14:15:20 my_table1 MYSQLItemNum 2000 # 2016-07-11 15:15:18 es_index1 ESItemNum 500
Step 3: Feed Data to ELK and Draw Useful Diagrams
To combine all the pieces together, here is a sample report. In the below example, the system uses CouchBase and Elasticsearch as Key DB stores.