DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Telemetry Pipelines Workshop: Installing Fluent Bit From Source
  • A Deep Dive Into Distributed Tracing
  • OpenTelemetry Moves Past the Three Pillars
  • Embracing the Future With Hybrid and Cloud-Native Observability: An In-Depth Exploration of Observability With Architectural Examples and Best Practices

Trending

  • My LLM Journey as a Software Engineer Exploring a New Domain
  • Breaking Bottlenecks: Applying the Theory of Constraints to Software Development
  • Hybrid Cloud vs Multi-Cloud: Choosing the Right Strategy for AI Scalability and Security
  • AI’s Role in Everyday Development
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Monitoring and Observability
  4. Getting Started With Prometheus Workshop: Relabeling Metrics in Prometheus

Getting Started With Prometheus Workshop: Relabeling Metrics in Prometheus

Interested in open-source observability? Learn about relabeling metrics and how to apply them when mapping, aggregating, dropping, or filtering scraped metrics.

By 
Eric D. Schabell user avatar
Eric D. Schabell
·
Eric D.  Schabell user avatar
Eric D. Schabell
DZone Core CORE ·
Jun. 16, 23 · Presentation
Likes (5)
Comment
Save
Tweet
Share
5.8K Views

Join the DZone community and get the full member experience.

Join For Free

Are you interested in open-source observability but lack the knowledge to just dive right in? This workshop is for you, designed to expand your knowledge and understanding of open-source observability tooling that is available to you today.

Dive right into a free, online, self-paced, hands-on workshop introducing you to Prometheus. Prometheus is an open-source systems monitoring and alerting tool kit that enables you to hit the ground running with discovering, collecting, and querying your observability today. Over the course of this workshop, you will learn what Prometheus is, what it is not, install it, start collecting metrics, and learn all the things you need to know to become effective at running Prometheus in your observability stack.

Previously, I shared an introduction to Prometheus, installing Prometheus, an introduction to the query language, exploring basic queries, and using advanced queries as free online workshop labs. In this article, you'll continue your journey with relabeling metrics in Prometheus.

Your learning path takes you into the wonderful world of relabeling in Prometheus, where you learn about aggregation, drop rules, and so much more. Note this article is only a short summary, so please see the complete lab found online here to work through it in its entirety yourself:


The following is a short overview of what is in this specific lab of the workshop. Each lab starts with a goal. In this case, it is the following:

The lab teaches you how relabeling metrics work in Prometheus and how to apply them when mapping, aggregating, dropping, or filtering scraped metrics. 

You start in this lab by ensuring the demo environment you learned about setting up in previous labs is running with two demo services and a single Prometheus instance. From there, you wander through the why and how of what relabeling is for in your metrics environment.

The basic layout of the configuration section is discussed before you embark on your first interactions with setting up static labels for your two demo services to use across the various exercises in this lab. They allow you to relabel without interfering with existing demo service labels and provide a baseline for a few use cases you'll work through later in the lab.

Your First Relabeling

A simple replacement action is the first exposure to relabeling in Prometheus, and it's done by setting an existing label value to a new fixed value. To facilitate the exercise, you are given a scenario as follows:

  • What if we decided for the upcoming vacation period, that team B would be the team to cover services support as owner while the rest are gone? We could set a temporary relabeling rule to replace all instances of the owner to team B. Imagine if you have thousands of services deployed needing this change? 


  • When the vacation is over, you just remove or turn off this relabeling rule.

Each of these exercises is structured the same. You are presented with the scenario, shown the structure of the relabeling action in question, walked through the structure so you understand it before using it, then shown how to implement the relabeling rule to solve the scenario. Below is the configuration and bold-faced marked relabeling rule for replacing all instances of the owner label found with teamB:

# Scraping services demo.
  - job_name: "services"
    static_configs:
        - targets: ["localhost:8080"]
	labels:
            service: demo1
            owner: teamD

	- targets: ["localhost:8088"]
	labels:
	    service: demo2
	    owner: teamA

    relabel_configs:

    # Relabeling owner for teamA vacation.
    - action: 'replace'
        replacement: 'teamB'
        target_label: 'owner'


After saving your configuration and restarting your Prometheus instance for the new configuration to take effect, you are then shown how to query the running instances to verify the label changes:

How to query the running instances to verify the label changes

After the vacation is over, you complete the exercise by removing your changes so that the proper owners of the services are again labeled correctly.

More Relabeling

The rest of the lab walks through many more relabeling actions with examples for aggregation, dropping labels, dropping objects, mapping, upper case, and lower case.

There are even more advanced use cases for relabeling addresses, such as HA environments that have a need for dropping metrics labels to prevent double alerts from firing. The following image is the result of applying aggregation relabeling to combine the owner and service metric labels into a new service_details metric label:

Result of applying aggregation relabeling to combine the owner and service metric labels into a new service_details metric label

You finish up this lab with an explanation of hashing and sharding of Prometheus instances and how hashmod relabeling helps you to target sharded services.

Missed Previous Labs?

This is one lab in the more extensive free online workshop. Feel free to start from the very beginning of this workshop here if you missed anything previously:


You can always proceed at your own pace and return any time you like as you work your way through this workshop. Just stop and later restart Perses to pick up where you left off.

Coming up Next

I'll be taking you through the following lab in this workshop where you'll start to explore how to auto-discover service targets for your metric scraping with Prometheus.

Stay tuned for more hands-on material to help you with your cloud-native observability journey.

Observability Open source Metric (unit) Aggregate data Cloud native computing

Published at DZone with permission of Eric D. Schabell. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Telemetry Pipelines Workshop: Installing Fluent Bit From Source
  • A Deep Dive Into Distributed Tracing
  • OpenTelemetry Moves Past the Three Pillars
  • Embracing the Future With Hybrid and Cloud-Native Observability: An In-Depth Exploration of Observability With Architectural Examples and Best Practices

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!