Scalable Metrics Storage: Gnocchi on AWS

DZone 's Guide to

Scalable Metrics Storage: Gnocchi on AWS

Considering a time series database? Gnocchi's DbaaS capabilities make it a solid option for cloud deployments. Here's how to use it to get your metrics up on AWS.

· Cloud Zone ·
Free Resource

As I wrote a few weeks ago in my post about Gnocchi 3.1 being released, one of the new features available in this version it the S3 driver. Today I would like to show you how easy it is to use it and store millions of metrics into the simple, durable, and massively scalable object storage provided by Amazon Web Services.

The installation of Gnocchi for this use case is not different than the standard installation procedure described in the documentation. Simply install Gnocchi from PyPI using the following command:

$ pip install gnocchi[s3,postgresql] gnocchiclient

This will install Gnocchi with the dependencies for the S3 and PostgreSQL drivers and the command-line interface to talk with Gnocchi.

Configuring Amazon RDS

Since you need a SQL database for the indexer, the easiest way to get started is to create a database on Amazon RDS. You can create a managed PostgreSQL database instance in just a few clicks.

Once you're on the homepage of Amazon RDS, pick PostgreSQL as a database:

You can then configure your PostgreSQL instance: I've picked a dev/test instance with the basic options available within the RDS Free Tier, but you can pick whatever you think is needed for your production use. Set a username and a password and note them for later: We'll need them to configure Gnocchi.

The next step is to configure the database in detail. Just set the database name to "gnocchi" and leave the other options to their default values (I'm lazy).

After a few minutes, your instance should be created and running. Note the endpoint. In this case, my instance is gnocchi.cywagbaxpert.us-east-1.rds.amazonaws.com.

In order to give Gnocchi an access to S3, you need to create access keys. The easiest way to create them is to go to IAM in your AWS console, pick a user with S3 access and click on the big gray button named "Create access key".

Once you do that, you'll get the access key id and secret access key. Note them down, we will need these later.

Creating gnocchi.conf

Now it's time to create the gnocchi.conf file. You can place it in /etc/gnocchi if you want to deploy it system-wide, or in any other directory, and add the --config-file option to each Gnocchi command.

Here are the values that you should retrieve and write in the configuration file:

  • indexer.url: the PostgreSQL RDS instance endpoint and credentials (see above) to set into.
  • storage.s3_endpoint_url: the S3 endpoint URL – that depends on the region you want to use and they are listed here.
  • storage.s3_region_name: the S3 region name matching the endpoint you picked.
  • storage.s3_access_key_id and storage.s3_secret_acess_key: your AWS access key id and secret access key.

Your gnocchi.conf file should then look like this:

url = postgresql://gnocchi:gn0cch1rul3z@gnocchi.cywagbaxpert.us-east-1.rds.amazonaws.com:5432/gnocchi [storage]
driver = s3
s3_endpoint_url = https://s3-eu-west-1.amazonaws.com
s3_region_name = eu-west-1
s3_access_key_id = <you access key id>
s3_secret_access_key = <your secret access key>

Once that's done, you can run gnocchi-upgrade in order to initialize the Gnocchi indexer (PostgreSQL) and storage (S3):

$ gnocchi-upgrade --config-file gnocchi.conf
2017-02-07 15:35:52.491 3660 INFO gnocchi.cli [-] Upgrading indexer 
2017-02-07 15:36:04.127 3660 INFO gnocchi.cli [-] Upgrading storage 

Then you can run the API endpoint using the test endpoint gnocchi-api and specify its default port 8041:

$ gnocchi-api --port 8041 -- --config-file gnocchi.conf
2017-02-07 15:53:06.823 6290 INFO gnocchi.rest.app [-] WSGI config used: /Users/jd/Source/gnocchi/gnocchi/rest/api-paste.ini
STARTING test server gnocchi.rest.app.build_wsgi_app
Available at
DANGER! For testing only, do not use in production

The best way to run the Gnocchi API is to use uwsgi as documented, but in this case, using the testing daemon gnocchi-api is good enough.

Finally, in another terminal, you can start the gnocchi-metricd daemon that will process metrics in the background:

$ gnocchi-metricd --config-file gnocchi.conf
2017-02-07 15:52:41.416 6262 INFO gnocchi.cli [-] 0 measurements bundles across 0 metrics wait to be processed.

Once everything is running, you can use Gnocchi's client to query it and check that everything is OK. The backlog should be empty at this stage, obviously.

$ gnocchi status
| Field                                               | Value |
| storage/number of metric having measures to process | 0     |
| storage/total number of measures to process         | 0     |

Gnocchi is ready to be used!

$ # Create a generic resource "foobar" with a metric named "visitor"
$ gnocchi resource create foobar -n visitor
| Field                 | Value                                         |
| created_by_project_id |                                               |
| created_by_user_id    | admin                                         |
| creator               | admin                                         |
| ended_at              | None                                          |
| id                    | b4d568e4-7af1-5aec-ac3f-9c09fa3685a9          |
| metrics               | visitor: 05f45876-1a69-4a64-8575-03eea5b79407 |
| original_resource_id  | foobar                                        |
| project_id            | None                                          |
| revision_end          | None                                          |
| revision_start        | 2017-02-07T14:54:54.417447+00:00              |
| started_at            | 2017-02-07T14:54:54.417414+00:00              |
| type                  | generic                                       |
| user_id               | None                                          |

# Send the number of visitor at 2 different timestamps
$ gnocchi measures add --resource-id foobar -m 2017-02-07T15:56@23 visitor
$ gnocchi measures add --resource-id foobar -m 2017-02-07T15:57@42 visitor

# Check the average number of visitor
# (the --refresh option is given to be sure the measure are processed)
$ gnocchi measures show --resource-id foobar visitor --refresh
| timestamp                 | granularity | value |
| 2017-02-07T15:55:00+00:00 |       300.0 |  32.5 |

# Now shows the minimum number of visitor
$ gnocchi measures show --aggregation min --resource-id foobar visitor
| timestamp                 | granularity | value |
| 2017-02-07T15:55:00+00:00 |       300.0 |  23.0 |

And voilà! You're ready to store millions of metrics and measures on your Amazon Web Services cloud platform. I hope you'll enjoy it and feel free to ask any question in the comment section or by reaching out to me directly!

amazon web services ,gnocchi ,metrics ,cloud storage ,cloud ,tutorial

Published at DZone with permission of Julien Danjou . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}