Monitoring Azure Activity Logs with Logz.io

In a previous post, we introduced a new integration with Microsoft Azure that makes it easy to ship Azure logs and metrics into Logz.io using a ready-made deployment template. Once in Logz.io, this data can be analyzed using the advanced analytics tools Logz.io has to offer — you can query the data, create visualizations and dashboards, and create alerts to get notified when something out of the ordinary occurs.

In this article, we'll take a look at how to collect and analyze a specific type of log data Azure makes available — Azure Activity Logs.

What Are Azure Activity Logs?

Simply put, Azure Activity Logs allow users to monitor who did what and when for any write operations (PUT, POST, DELETE) executed for Azure resources in a specific Azure subscription and to understand the status of the operation and other relevant properties. You can, for example, use Activity Logs to gain insight into when new VMs are created, updated, or deleted via the Resource Manager.

There are several different categories of Activity Logs, each giving you a different type of insight into what is transpiring within your subscription — Administrative, Service Health, Resource Health, Alert, Autoscale, Recommendation, Security, and Policy. To understand the different types of Activity Log categories, I recommend Azure's docs on the topic.

Deploying the Template

First, you will need to deploy the template (if you've already set up the integration with Logz.io, feel free to skip to the next step). The easiest way to do this to use the Deploy to Azure button displayed in the first step of the repo's readme:

Once clicked, the Custom Deployment page in the Azure portal will be displayed with a list of pre-filled fields.

You can leave most of the fields as-is but be sure to enter the following settings:

• Resource group: Either select an existing group or create a new one.
• Logzio Logs Host: Enter the URL of the Logz.io listener. If you're not sure what this URL is, check your login URL — if it's app.logz.io, use listener.logz.io (this is the default setting). If it's app-eu.logz.io, use listener-eu.logz.io .
• Logzio Metrics Host: Enter the URL of the Logz.io listener. If you're not sure what this URL is, check your login URL — if it's app.logz.io, use listener.logz.io (this is the default setting). If it's app-eu.logz.io, use listener-eu.logz.io.
• Logzio Logs Token: Enter the of the Logz.io account you want to ship Azure logs to. You can find this token on the account page in the Logz.io UI.
• Logzio Metrics Token: Enter a token for the Logz.io account you want to use for shipping Azure metrics to. You can use the same account used for Azure logs.

Agree to the terms at the bottom of the page, and click Purchase.

Azure will then deploy the template. This may take a while as there is a long list of resources to be deployed, but after a minute or two, you will see the Deployment succeeded message at the top of the portal.

Streaming Azure Activity Logs to Logz.io

Now that we have all the building blocks in place for streaming the data into Logz.io, our next step is to set up exporting activity logs.

Activity logs can be exported to Events Hub, which fits our scenario perfectly.

Open the Activity Log in the Azure portal and click Export to Event Hub at the top of the page.

In the Export activity log blade that's displayed, select Export to an event hub and then click Select a service bus namespace.

Enter the details of the Logz.io event hub namespace and policy name, and click OK.

Save the settings.

Azure will apply the settings, and within a minute or two, you will start to see activity logs in Logz.io.

Analyzing Azure Activity Logs

Azure Activity Logs contain a wealth of information that can be used for tracking activities within a subscription. There are various categories of events recorded in this data, each with a different set of fields available for analysis.

To begin your analysis in Logz.io, you will most likely start with the Discover page in Kibana. Start by selecting some fields from the list on the left to get more visibility into the data. For example, in the example below, I added the operationName and durationMs fields:

Using different types of queries, you can then search for specific events.

To examine only write events, for example, use:

category:Write        

Or, say you want to find write actions performed within a specific Azure region:

category:Write AND location:westus

Kibana supports rich querying options that will help you dive deeper into the rabbit hole. 

Visualizing Azure Activity Logs

Of course, Kibana is well known for its visualization capabilities, and once you've gained a better understanding of the data collected in Activity Logs, you can start building visualizations. Again, there is a wide variety of options to play around with and I'll provide you with some examples here.

Operation Type Breakdown

The category field details the operation type — "Write", "Delete," or "Action. Using a pie chart visualization, we can monitor this breakdown to get a picture of the different operations performed in our Azure subscription.

Locations Breakdown

In a similar fashion, we can monitor operations across regions — this time using the field:

Status Codes Over Time

The Azure Activity Log also reports the status for executed operations, such as "Started," "Created," "Active," etc. Using a bar chart visualization, we can see a breakdown of these codes over time.

Avg. Action Duration

The durationMs informs us how long the different actions take to execute. Line chart visualizations are great for monitoring trends over time, so we can use an average aggregation of this field to get an overview picture of our Azure actions:

Activities per User

Another example is listing activities per user. One way of visualizing this data is by using a data table visualization:

Adding all your visualizations into a dashboard gives you a nice overview of all the activity being recorded in Azure's Activity Log.

The dashboard above is available for one-click deployment in ELK Apps — Logz.io's library of pre-made dashboards and visualizations. To deploy, simply open ELK Apps, search for Azure, and hit the Install button.

Endnotes

The Activity Log is a great way to keep track of the different operations being executed by users in your Azure subscriptions. It provides details on who did what, when and in what region. The integration with Logz.io adds advanced analysis capabilities on top of this data.

As mentioned, Azure also generates diagnostic logs that together with the Activity Log gives you a comprehensive view of your Azure environment. To find out more about shipping and analyzing Azure Monitor logs and metrics, take a look at Monitoring Azure with Logz.io.