DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • Monitoring Apache Ignite Cluster With Grafana (Part 1)
  • Revolutionizing Observability: How AI-Driven Observability Unlocks a New Era of Efficiency
  • A Deep Dive Into Distributed Tracing
  • 3 Best Tools to Implement Kubernetes Observability

Trending

  • MCP Servers: The Technical Debt That Is Coming
  • Optimizing Integration Workflows With Spark Structured Streaming and Cloud Services
  • Can You Run a MariaDB Cluster on a $150 Kubernetes Lab? I Gave It a Shot
  • Unlocking Data with Language: Real-World Applications of Text-to-SQL Interfaces
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Monitoring and Observability
  4. Getting Started With Prometheus Workshop: Exploring Basic Queries

Getting Started With Prometheus Workshop: Exploring Basic Queries

In this tutorial, you'll continue your open-source observability journey exploring basic Prometheus queries using PromQL.

By 
Eric D. Schabell user avatar
Eric D. Schabell
·
Eric D.  Schabell user avatar
Eric D. Schabell
DZone Core CORE ·
Apr. 15, 23 · Tutorial
Likes (3)
Comment
Save
Tweet
Share
8.1K Views

Join the DZone community and get the full member experience.

Join For Free

Are you looking to get away from proprietary instrumentation?  Are you interested in open-source observability, but lack the knowledge to just dive right in? This workshop is for you, designed to expand your knowledge and understanding of open-source observability tooling that is available to you today.

Dive right into a free, online, self-paced, hands-on workshop introducing you to Prometheus. Prometheus is an open-source systems monitoring and alerting tool kit that enables you to hit the ground running with discovering, collecting, and querying your observability today. Over the course of this workshop, you will learn what Prometheus is, what it is not, install it, start collecting metrics, and learn all the things you need to know to become effective at running Prometheus in your observability stack. 

Previously, I shared an introduction to Prometheus, installing Prometheus, and an introduction to the query language as free online workshop labs. In this article, you'll continue your journey exploring basic Prometheus queries using PromQL.

Your learning path continues in this article with the exploration of a few basic PromQL queries. Note this article is only a short summary, so please see the complete lab found online here to work through it in its entirety yourself:

The following is a short overview of what is in this specific lab of the workshop. Each lab starts with a goal. In this case, it is fairly simple:

  • This lab dives into using PromQL to explore basic querying so that you can use it to visualize collected metrics data.

You'll start off by getting comfortable selecting metrics and at the same time learning the basic definitions of metrics query terminology. Each query is presented for you to cut and paste into your own Prometheus expression browser and followed by an example output so that you see what the query is doing. 

These examples have been generated on a system running for several hours giving them more color and diversity when graphing data results that you might see if you are quickly working self-paced through this workshop. This doesn't take away from the exercise; it's meant to provide a bit more extensive example results than you might be seeing yourself.

Next, you'll learn about how to filter your query results as you narrow the sets of data you're selecting using metric labels, also known as metric dimensions. You'll explore matching operators, filtering with regular expressions, dig into instant vectors, range vectors, learn about time-shifting, explore a bunch of ways to visualize your data in graphs, take a side step into the discussion around the functions without and by, and finally, learn how to use math in your queries.

Selecting Data

The very basic beginning of any query language is being able to select data from your metrics collection. You'll kick this lab off learning about the basic terminology involved with your metrics queries, such as:

  • Metric name - Querying for all time series data collected for that metric
  • Label - Using one or more assigned labels filters metric output
  • Timestamp - Fixes a query single moment in the time of your choosing
  • Range of time - Setting a query to select over a certain period of time
  • Time-shifted data - Setting a query to select over a period of time while adding an offset from the time of query execution (looking in the past)

Applying all of these you'll quickly start selecting metrics by name, such as the metric below from the services demo you set up in a previous lab:

demo_api_request_duration_seconds_count


This query, when entered in your Prometheus expression browser will result in something like the following query results:

Query results

You'll continue to learn how to filter using labels, and how to narrow down your query results and metrics dimensions. After several iterations,  you'll eventually find the data you're looking for with the following selection query filtering with multiple labels:

demo_api_request_duration_seconds_count{method="POST", path="/api/foo", status="500"}


The final results are much more refined than where you started:

Refined final results

This wraps up the selection query section and sends you onward to having hands-on practical experience with selection queries.

More Ways to Skin Your Data

While using the equals operator is one way to select and filter data, there are more ways to approach it. The next section spends some time sharing how you can use regular expressions in your queries. This wraps up with you learning that up to now you've done queries selecting the single latest value for all time series data found, better known as an instant vector. 

You are now introduced to the concept of a range vector, or queries using functions that require a range of values. They have a duration specifier in the form of [number, unit]; for example, the following query selects all user CPU usage over the last minute:

demo_cpu_usage_seconds_total{mode="user"}[1m]


Resulting in the following display of all the values found over the last minute for this metric:

Values found over the last minute in this metric

Next, you'll learn how to take the range vectors and apply a little time-shifting to look at a window of data in the past, something that is quite common in troubleshooting issues after the fact.

While this can be fun, one of the most common things you'll want to see in your data is how some metric has changed over a certain period of time. This is done using the rate function and you'll explore both this and its helper, the irate function. Below is the rate query:

rate(demo_api_request_duration_seconds_count{method="POST"}[1m])


And the corresponding results in a graph:

Results graph

While these functions up to now have visualized counters in graph form, you have many other types of metrics you'll want to explore, such as gauge metrics. You'll spend time exploring all these examples and running more queries to visualize gauge metrics.

This leads to a bit of a more complex issue. What do you do with queries where you want to aggregate over highly dimensional data in your queries? This requires simplifying and creating a less detailed view of the data results with functions like sum, min, max, and avg. These do not aggregate over time, but across multiple series at each point in time. 

The following query will give you a table view of all the dimensions you have at a single point in time with this metric:

demo_api_request_duration_seconds_count{job="services"}


The results are showing that this metric is highly dimensional:

Results showing highly dimensional metric

In this example, you are going to use the sum function, look across all the previous dimensions for a five-minute period of time, and then look across all captured time series data:

sum(
  rate(demo_api_request_duration_seconds_count{job="services"}[5m])
)


The resulting graph shows the sum across highly dimensional metrics:

Graph showing the sum across highly dimensional metrics

You'll work through the entire list of aggregation functions, testing most of them in hands-on query examples using your demo installation to generate visualizations of your time series data collection.

You'll finish up this lab by learning about how to apply arithmetic, or math to your queries. You'll sum it all up with a query to calculate per-mode CPU usage divided by the number of cores to find a per-core usage value of 0 to 1. To make it more interesting, the metrics involved have mismatched dimensions, so you'll be telling it to group by the one extra label dimension. 

It all sounds like a complex problem you have no idea how to solve? Don't worry, this entire lab builds you from the ground up to reach this point. You will be running this query to solve this problem and ready for the next more advanced query lab coming up!

Missed Previous Labs?

This is one lab in the more extensive free online workshop. Feel free to start from the very beginning of this workshop here if you missed anything previously:

You can always proceed at your own pace and return any time you like as you work your way through this workshop. Just stop and later restart Perses to pick up where you left off.

Coming Up Next

I'll be taking you through the following lab in this workshop where you'll continue learning about the PromQL and dig deeper into advanced queries to gain more complex insights into your collected metrics. Stay tuned for more hands-on material to help you with your cloud-native observability journey.

Observability Open source Time series Data (computing) Metric (unit)

Published at DZone with permission of Eric D. Schabell. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Monitoring Apache Ignite Cluster With Grafana (Part 1)
  • Revolutionizing Observability: How AI-Driven Observability Unlocks a New Era of Efficiency
  • A Deep Dive Into Distributed Tracing
  • 3 Best Tools to Implement Kubernetes Observability

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!