Exporting Logentries Data With Leexportpy
Exporting Logentries Data With Leexportpy
Without any modification to the current code, you can use services such as Kafka, Geckoboard and Hosted Graphite to extract your Logentries data.
Join the DZone community and get the full member experience.Join For Free
DevOps involves integrating development, testing, deployment and release cycles into a collaborative process. Learn more about the 4 steps to an effective DevSecOps infrastructure.
Leexportpy, the Logentries utility that exports your log data to 3rd parties, has built-in support for various services such as Kafka, Geckoboard and Hosted Graphite. Without any modification to the current code, you can use these services to extract your Logentries data.
To begin, make sure your read-write or read-only API key is correctly placed in the LE section of your configuration file as shown below. Also make sure you have read the first blog post of this series.
[LE] ro_api_key = <your Logentries read-only api key> rw_api_key = <your Logentries read-write api key>
Leexportpy provides easy to use support for appending transformed Logentries statistics data to a Kafka topic. The current implementation of the Kafka Service is a great reference for showing the querying, transformation and export of Logentries data to a Kafka topic using Leexportpy. The data extracted from Logentries is based on querying your log data using a LEQL query, the results, which are either time series, or groupby statistics is then appended to Kafka topic. The structure of the service and the destination configuration represents what a typical Leexportpy Kafka service should look like.
[[[destination]]] service = kafka brokers = <comma separated host:port of brokers e.g: “localhost:9092,localhost:9093”> topic = <name of the kafka topic>
Statistics data from a query will be appended to Kafka Topic in the order they received in the response, for example see the images below:
We support various Geckoboard custom widgets in leexportpy. To export Logentries data to Geckoboard, you should include a “geckoboard” entry in the services section of your configuration file along with your Geckoboard API key, in this format:
[Services] [[geckoboard]] api_key = <your geckoboard api key>
This format, as you will see in other services, applies to all used services even if no API key is needed for that service. Destination services that are used by searches should be present in the Services section with the same service name(service) in the destination configuration.
Some of the currently supported geckoboard widgets are highlighted below along with the corresponding destination configuration that the related Search should have.
Note: To get the push URL for your custom widget, in the Geckoboard UI go to edit page of your widget and click “Push” in the “Method” section. You can also get the push URL during the creation of a widget.
The Line Chart service of Leexportpy expects a timeseries (non groupby) query response. The configuration of a line chart widget should look like:
[[[destination]]] push_url = <push url for line chart widget> service = geckoboard widget_type = line_chart name = <name of your series in line chart>
The query that provides data to Bar Chart can be either timeseries or groupby. The configuration of a bar chart widget should look like:
[[[destination]]] push_url = <push url for bar chart widget> service = geckoboard widget_type = bar_chart
The Pie Chart expects groupby query response to transform and push data to Geckoboard. The configuration of a pie chart widget should look like:
[[[destination]]] service = geckoboard push_url = <your geckoboard push url for pie chart widget> widget_type = pie_chart
Number and Secondary Stat:
The Number and Secondary Stat widget expects time series (non groupby) query because global count can be found only on timeseries queries.
[[[destination]]] service = geckoboard push_url = <push url for number stat widget> widget_type = number_stat text = <name of your number stat>
Hosted Graphite is another 3rd party service currently supported in Leexportpy. You can now export your Logentries timeseries data for processing to Hosted Graphite.
To setup Hosted Graphite integration the hosted_graphite section with a valid Hosted Graphite API key in the Services section of your configuration file as shown below.
[Services] [[hosted_graphite]] api_key = <your Hosted Graphite API key>
Note: Only timeseries query results will be pushed to hosted graphite. Timestamps in the response will be pushed to Hosted Graphite along with their corresponding “count” values. The destination format of your search should be this:
[[[destination]]] push_url = https://www.hostedgraphite.com/api/v1/sink service = hosted_graphite metric_name = <metric name in the hosted graphite>
Currently supported services are production ready and provide examples of what can be done with the data provided by the Logentries REST Query API and the scheduling & processing mechanisms of Leexportpy. What you can do with Leexportpy is not limited to the currently supported services. Any service that is capable of accepting and processing your transformed log data can be implemented as a service in Leexportpy and give you the ability to synchronize and process your Logentries data.
In the following and last blog post of this series we will be showing how to create a new service from scratch and synchronizing our log data with a new external system.
Published at DZone with permission of Safa Topal , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.