Monitoring Couchbase Sync Gateway With Prometheus and Grafana
Learn how to successfully setup monitoring with Prometheus and Grafana and drive replications with Couchbase Lite clients and monitor it.
Join the DZone community and get the full member experience.Join For Free
The Couchbase Mobile 2.5 release introduced extensive stats reporting capabilities on the Sync Gateway.The stats provide key insights into the health of your Couchbase Mobile deployment and constitues an integral part of any deployment.
In this post, we discuss how you can use Prometheus, an open source monitoring and alerting platform for monitoring your Sync Gateway nodes and Grafana for visualizing the stats. In an upcoming related post, we will discuss how you can setup monitoring with Prometheus on a Couchbase Mobile Kubernetes cluster.
Sync Gateway Stats Reporting
Prometheus is an open source systems monitoring and alerting platform and hosted by Cloud Native Computing Foundation. At the core of it is the Prometheus Server that is responsible for polling "Prometheus targets" for stats and storing it as time series data. Prometheus targets are statically configured or can be discovered by Prometheus.
Grafana is an open source data visualization and alerting platform. It supports Prometheus as a data source and can be used to build comprehensive dashboards.
Introducing the Prometheus Sync Gateway Exporter
In order for Prometheus to monitor the Sync Gateway, we need a "Prometheus target" corresponding to the Sync Gateway. This target is the Sync Gateway Exporter, herein referred to as "the Exporter". Simply put, the Exporter is responsible for exporting Sync Gateway stats to Prometheus metrics.
Tying all this together, a typical deployment would like the following -
- Sync Gateway Exporter The Sync Gateway Admin REST API is, by default, only exposed on localhost. This configuration is strongly recommended in production enviornments as well. As a result, since the Exporter polls the Sync Gateway Admin API for stats, the Exporter would need to be on the same host/node as the Sync Gateway. Also, the Sync Gateway stats are reported on a per-node basis. So in a cluster of two or more Sync Gateway nodes, each Sync Gateway node must have it's own Exporter.
- Prometheus Server continually polls the Exporter for stats, which in turn polls the Sync Gateway REST endpoint. The server uses the rules defined in the rules.yaml to push alerts to the Alerting Manager.
- Grafana service polls the Prometheus Server for stats and graphs it on a web-based dashboard that can be accessed via the web browser.
In the remainder of the post, we will walk through the steps to setup a Couchbase Mobile cluster for monitoring with the Exporter, Prometheus Server and Grafana. While the instructions are intended for a development environment, you can easily customize these for a production environment.
We will be using Docker, which will make it extremely simple to get up and going.
The Sync Gateway Exporter is open source couchbaselabs project and available on GitHub. It is compatible with Couchbase Sync Gateway 2.5. To that end, while we will strive to keep it up-to-date with the latest release of Sync Gateway, we cannot guarantee that. But the good news is that it is open source and includes detailed instructions on how you can extend it to support additional stats.
Download the Exporter source from GitHub.
git clone https://github.com/couchbaselabs/couchbase-sync-gateway-exporter
Deploying Couchbase Mobile Cluster
All commands in this section must be executed from the root folder of the cloned GitHub repo.
If you already have a Couchbase Mobile cluster running on Docker, you can skip this step and move on to the "Deploying Sync Gateway Exporter" step. Just remember to substitute the Docker network name corresponding to your existing setup in the instructions below.
Creating a Docker Network
When using docker, it is recommended to run all the components on same docker network.
Create a docker network with the name "demo."
docker network create demo
Deploying a Couchbase Server
We will use a special Docker image of the Couchbase Server v6.0.1 that includes a preinstalled sample bucket called "TravelSample" as well as a preconfigured RBAC user with "Application Access" that would enable Sync Gateway to connect to the Couchbase Server.
docker run -d –name cb –network demo -p 8091–8094:8091–8094 -p 11210:11210 connectsv/server:6.0.1-enterprise
Deploying Sync Gateway
The Sync Gateway will be launched with the config file named sync-gateway-config.json that is located in the "testdata" folder of the cloned repo.
docker run -p 4984–4985:4984–4985 –network demo –name sync-gateway -d -v <code>pwd</code>/testdata/sync-gateway-config.json:/etc/sync_gateway/sync_gateway.json couchbase/sync-gateway:2.5.0-enterprise -adminInterface :4985 /etc/sync_gateway/sync_gateway.json
Try it out.
Confirm that your Sync Gateway is up and running and that the
_expvar endpoint is accessible by running the following
curl command. If things are working as expected, you should see a whole lot of stats output to the console.
curl GET http://localhost:4985/_expvar
Deploying Sync Gateway Exporter
The Docker image of the Sync Gateway Exporter is available at
sgw.url config option is of particular interest. This points to the Sync Gateway server. If you are working with an existing deployment of Couchbase Mobile, make sure you set the
sgw.url to point to the Sync Gateway node in your setup.
The Exporter binds to
0.0.0.0 by default. You can override that using
docker run -p 9421:9421 –network demo –name exporter -d couchbasesamples/sync-gateway-prometheus-exporter:latest –log.level=debug –sgw.url=http://sync-gateway:4985
Try it Out
Confirm that the exporter is up and running with the following
If things are working, there will be a bunch of metrics output to the console. Confirm that the "sgw_up 1".
Deploying Prometheus Server
Now that we have the Exporter exporting stats from the Sync Gateway, we will setup Prometheus to poll the Exporter for metrics.
Exploring the prometheus.yml File
prometheus.yml file specifies the config with which the Prometheus server is launched. It is located in the "testdata" folder of the cloned repo. Let's examine the contents of this file.
global: scrape_interval: 5s evaluation_interval: 5s
rule_files: - &apos;/etc/prometheus/rules/*&apos; scrape_configs: - job_name: swg static_configs: - targets: - exporter:9421</code></pre>
scrape_intervalspecifies the polling interval. You can adjust it per your needs.
rule_filesspecifies the location of the Prometheus Rules file. The Prometheus rules file,
sync-gateway.rules.ymlis located in the
testdata/rules/folder in the cloned repo. The rules are used by Prometheus to trigger alerts. We have provided a baseline set of rules as a starting point. You can customize the rules as needed. The rules file is mounted at
/etc/prometheus/ruleswhen the Docker container is launched.
targetsspecifies the Prometheus targets which is the Exporter. You could have multiple exporters if you have multiple Sync Gateways. In that case, you can statically specify all the Exporter endpoints or configure Prometheus to discover the targets.
Installing Docker Image
We will deploy the official docker image
prom/prometheus with the
prometheus.yml file included in the cloned repo.
docker run -p 9090:9090 –network demo –name prometheus -d -v <code>pwd</code>/testdata/prometheus.yml:/etc/prometheus/prometheus.yml -v <code>pwd</code>/testdata/rules:/etc/prometheus/rules prom/prometheus
Try it Out
Open the URL
http://localhost:9090/graph in a web browser.
Click on the "Insert metric at Cursor" button to see list of available Sync Gateway metrics.
Click on "Status"->"Targets" to see the list of exporters and their stats.
Click on "Alerts" menu button to see the status of the alerts.
While you could use Prometheus's web UI for visualizing the stats, we will be using Grafana as you can create some very compelling dashboards with it and Grafana integrates well with Prometheus.
- We will install the official
grafana/grafana:6.2.0docker image. The
grafana/datafolder in the cloned repo contains the
grafana.dbthat is used for persisting the dashboard and other metadata. This is volume mounted at
docker run -p 3000:3000 –network demo –name grafana -d grafana/grafana:6.2.0 -v <code>pwd</code>/grafana/data:/var/lib/grafana
- A default “Couchbase Sync Gateway” dashboard called
dashboard.jsonnetis available in the cloned repo as a jsonnet file. Make the
grafanatarget to generate the corresponding
dashboard.jsonfile that can then be imported into Grafana. The dashboard graphs all the metrics exported by the Exporter. You can customize it per your needs.
# Pull in relevant submodules that jsonnet needs git submodule update –init –rebase –remote –recursive # Make the grafana target to generate the dashboard.json file make grafana
Try it Out
- (Re)Generates the
dashboard.jsondashboard from the
- Uses Grafana API to add Prometheus as data source
- Uses Grafana API to load the
dashboard.jsongenerated in the previous step
- Click on the "Couchbase Synce Gateway Dashboard" to see the stats. You can filter it by Sync Gateway (ifyou have more than one) or database.
That's it! You have successfully setup monitoring with Prometheus and Grafana. You can now drive replications with Couchbase Lite clients and monitor it. The default Sync Gateway dashboard is a starting point. You can customize the dashboard, either by editing the
dashboard.jsonnet file or directly through the Grafana UI.
Instead of running each container independently, you can do a multi-container launch using docker-compose.
The repo includes a
docker-compose.yml file that you can use to launch all the containers including Couchbase Server, Sync Gateway, Exporter, Prometheus and Grafana. There are some caveats. You can learn more about it here.
Using the Sync Gateway Exporter, you can export Sync Gateway stats to Prometheus and visualize it using visualization tools such as Grafana. This vastly simplifies the monitoring of your Couchbase Mobile clusters. In addition to Exporter, we have provided a default Grafana dashboard that you can customize. Customizing the dashboard itself, you can also customize the rules for alerts. While we discussed the installation using Docker containers, you could also install the binaries directly.
I would like to thank Carlos Becker for his work on building the first version of the Sync Gateway Exporter.
Published at DZone with permission of Priya Rajagopal, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.