Introduction to Leexportpy

What Is Leexportpy?

Leexportpy is a flexible and extensible Python application that enables log search results to be easily exported to third party services such as Geckoboard and HostedGraphite. The application utilizes the Logentries REST API providing you with the ability to easily create queries on any of your log data, transform the results to the data format needed by the third party service, and push transformed data to that service. In other words, it acts as a flexible search tool and proxy between the Logentries logging engine and third party services.

The current version of Leexportpy supports integration with the data visualization tools Geckoboard and Hosted Graphite. In a follow-up blog post in this series, we will show how to easily add your own plugin to push Logentries query results to any service or endpoint you desire.

What Does It Do?

Leexportpy takes LEQL queries specified in the configuration file, uses the Logentries REST API to obtain the query results and converts the query result to the data structure accepted by the third party service, and then pushes the transformed result to a specified endpoint. Now, imagine you can do this whole task periodically in configurable intervals. Then add the possibility of having multiple search and push tasks that can run concurrently. This is what Leexportpy does.

The configuration file is used to define all of the parameters used to setup Leexportpy. This includes your Logentries REST API keys, log keys of the logs you wish to query, the set of LEQL queries you wish to run, the query time ranges, and the third party service settings such as the push endpoints and visualization widget settings.

Configuring Leexportpy

Leexportpy accepts a configuration file that contains structured configuration data like the Python library ‘ConfigParser’ (also similar to how Windows .INI files are structured) and additionally it supports nested configuration sections. We implemented nested configuration sections for better support of multiple services and multiple searches in a single instance of Leexportpy.

One of the features of Leexportpy is to reload the configuration file every five minutes. This means that the configuration file is re-read which makes sure the configuration and search tasks are up-to-date. Therefore, the user does not have to restart the application after every configuration change.

Each section in the configuration file will be explained below showing the minimal requirements for a simple export.

An example configuration file is provided here:

[LE]
    rw_api_key = <Your read-write Logentries API key> # if defined, rw_api_key will be used for all queries.
    ro_api_key = <Your read-only Logentries API key> # if rw_api_key is not defined, ro_api_key will be used.

[Services] # this is the list of services you want to push your data.
    [[geckoboard]]
        api_key = <Your geckoboard API key>

    [[hosted_graphite]]
        api_key = <Your hostedgraphite API key>

    [[dummy]]
        api_key = <An API key for dummy service>

[Searches]
    [[Search0]] # these section names should be unique, other than that they don't affect the functionality.
        [[[query]]]
            logs = <Your logkey to query>
            query_period = <Query & push period in seconds>
            statement = <Your leql query to be run>
            query_range = <The time range of the query in seconds - (the time now minus query_range will be the beginnin time of your query)>
        [[[destination]]]
            push_url = <URL to push transformed data>
            service = <Service name that should match the defined one in Services section above>
            custom_config = <Custom config that is specific to this service/destination>

    [[Search1]]
        [[[query]]]
            ...
        [[[destination]]]
            ...

[LE]

This is the authentication section for the Logentries API keys used to access the query REST API. Leexportpy supports two kinds of API keys to authenticate, a read-only and read-write key, both of these key types can be used to query data on the Logentries REST API:

1. Read-write API key.
   • rw_api_key = your Logentries read-write API key
2. Read-only API key.
   • ro_api_key = your Logentries read-only API key

To obtain your Logentries API keys, go to the account section of the Logentries application and select the API Keys tab.

Note: If both keys are defined in the configuration file, read-write API key will be preferred.

[Services]

In this section, you define the service specific configurations/settings for the third-party services with which you wish to integrate, the searches used to generate the exported data is configured in a later [Searches] section. The [Services] section of the configuration file should include parameters such as the API keys for the third party service push APIs. For example, if you will be using Hosted Graphite and your Service key for Hosted Graphite is: hosted_graphite. Then the [Services] section should look like this:

[Services]
    [[hosted_graphite]]
        api_key = your Hosted Graphite API key.
    [[geckoboard]]
        api_key = your geckoboard API key.

Currently supported 3^rd party services and their service keys are:

1. Geckoboard, key string in configuration file: geckoboard
2. Hosted Graphite, key string in configuration file: hosted_graphite

We will be talking about how we define our service keys based on the real name of the service in later blog posts. For detailed information on this, check Adding a New Service section in the documentation.

[Searches]

This section is used to define the LEQL based searches that we are going to run periodically. Here we define every query and destination configuration of our searches.

First, we define a subsection with a unique name of our Search. This uniqueness is necessary for the correct parsing of configuration file only.

For the sake of this example, let’s call it: MyUniqueSearchName

[[MyUniqueSearchName]]

Every search has two mandatory subsections: query and destination. This is where we define every aspect of our search. Every configuration here is mandatory.

[[query]]
      logs = your log-key
      query_period = query period for search
      statement = leql query to run
      query_range = your query range up to now() in seconds
[[destination]]
      push_url = 3rd party url to push transformed data
      service = name of the service
      service_specific_custom_key = (See below)


service_specific_custom_key is a custom config specific to this service, i.e: for geckoboard, we should be inserting a config key here called: “widget_key” as currently geckoboard needs different data transformation for its different widgets

Installing Leexportpy

Install by providing the GitHub repository URL:

pip install <url_of_repository>

If you have already downloaded the source code to your local machine go to root directory of the project and run:

pip install .

Uninstalling Leexportpy

pip uninstall leexportpy

Running Leexportpy

After a successful installation of Leexportpy, you can control Leexportpy with the command leexportpy. There are two command groups Leexportpy supports: start and stop

Start

Leexportpy can be run in both background and foreground modes:

Foreground mode redirects all logs to standard output (stdout) and leaves daemonizing the process to the user. This mode can be used without sudo privileges.

leexportpy start -c <your_config_file>

Background mode runs leexportpy as a daemon. This should be done with root privileges as it needs to create a PID file and a log file in directories that require root permissions. The PID file is located at /var/run/leexportpy.pid and the log file is located at /var/log/leexportpy.log

sudo leexportpy start -c

<absolute_path_to_your_configuration_file> -d 

Note: If trying to run it in daemonized mode, absolute path of your configuration file should be provided.

Stop

If leexportpy is running in the background (daemonized), you will need sudo privileges as it’s necessary to read the PID file of our process:

sudo leexportpy stop

If running in the foreground

CTRL + C

Conclusion

Leexportpy gives you the ability to query, transform and transfer your Logentries data to a third party or your own services. Leexportpy is portable, extremely flexible and extensible, meaning there are a broad range of use cases. It integrates out of the box with HostedGraphite and Geckoboard. With small additional service implementation, endpoints that the user is pushing data to can vary from REST APIs, messaging queues, notification systems, databases and so on. Leexportpy enables you to get even more business value from your Logentries data, providing you with an easy way to extract search results for further analysis and integration with other data analysis and visualization tools.

The next blog posts in this series will cover details of the currently supported services such as Geckoboard and Hosted Graphite, and how to develop your own service plugin for Leexportpy, stay tuned!