{{announcement.body}}
{{announcement.title}}

How to Use Azure Go SDK to Manage Azure Data Explorer Clusters

DZone 's Guide to

How to Use Azure Go SDK to Manage Azure Data Explorer Clusters

In this article, see how to programmatically manage Azure Data Explorer resources using Go.

· Big Data Zone ·
Free Resource

Getting started with Azure Data Explorer using the Go SDK covered how to use the Azure Data Explorer Go SDK to ingest and query data from azure data explorer to ingest and query data. In this blog you will the Azure Go SDK to manage Azure Data Explorer clusters and databases.

Azure Data Explorer (also known as Kusto) is a fast and scalable data exploration service for analyzing large volumes of diverse data from any data source, such as websites, applications, IoT devices, and more. This data can then be used for diagnostics, monitoring, reporting, machine learning, and additional analytics capabilities.

In case you're wondering, we are talking about two different SDKs here. The one covered in this blog is for resource administration (also known as the control plane SDK) and the the one I used in the other post is data plane SDK for interacting with the Azure Data Explorer service itself (ingestion, query etc.)

What's Covered?

A simple CLI application is used as an example to demonstrate how to use the Go SDK. We'll try out the application first and go through how to:

  • Create and list Azure Data Explorer clusters
  • Create and list databases in that cluster
  • Delete the database and cluster
Java
 




x
15


 
1
$ ./goadx --help
2
CLI to test sample program for Azure Data Explorer
3
 
          
4
Usage:
5
  goadx [command]
6
 
          
7
Available Commands:
8
  cluster     create, list and delete Azure Data Explorer clusters
9
  db          create, list and delete databases in an Azure Data Explorer cluster
10
  help        Help about any command
11
 
          
12
Flags:
13
  -h, --help   help for goadx
14
 
          
15
Use "goadx [command] --help" for more information about a command.



Once that's done, we'll walk through the sample code to understand what's going on

The code is available on GitHub https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer

Please note that this CLI based example is just meant to showcase how to use the Azure Go SDK (in the context of Azure Data Explorer) as a part of a larger application. It is not supposed to replace/substitute the Azure CLI which can be used to manage Azure Data Explorer resources

Pre-requisites

Install Go 1.13 or above

You will need a Microsoft Azure account. Go ahead and sign up for a free one!

Install the Azure CLI if you don't have it already (should be quick!)

Run the CLI Application

Get the code and build it:

Java
 




xxxxxxxxxx
1


 
1
git clone https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer
2
cd azure-go-sdk-for-dataexplorer
3
go build -o goadx
4
 
          
5
//to confirm
6
chmod a+x goadx && ./goadx



To see the details for individual commands, e.g. for cluster creation:

Java
 




xxxxxxxxxx
1
17


 
1
./goadx cluster create --help
2
 
          
3
//output
4
 
          
5
Creates 1 instance of compute type DevNoSLAStandardD11V2 in Basic tier
6
 
          
7
Usage:
8
  goadx cluster create [flags]
9
 
          
10
Flags:
11
  -h, --help          help for create
12
      --loc string    ADX cluster location
13
      --name string   ADX cluster name
14
 
          
15
Global Flags:
16
      --rg string    Azure resource group
17
      --sub string   Azure subscription


A Few More Steps Before Creating the Cluster...

Create a resource group...

...using az group create CLI command

Java
 




xxxxxxxxxx
1


 
1
az group create -l <region> -n <name>
2
 
          
3
e.g. az group create -l southeastasia -n my-adx-rg



We need a Service Principal for the Go SDK to authenticate with Azure Data Explorer service to execute cluster and database operations.

Create a Service Principal...

...using az ad sp create-for-rbac

Java
 




xxxxxxxxxx
1


 
1
az ad sp create-for-rbac -n "test-datax-sp"



You will get a JSON response as such

Java
 




xxxxxxxxxx
1


 
1
{
2
  "appId": "fe7280c7-5705-4789-b17f-71a472340429",
3
  "displayName": "test-datax-sp",
4
  "name": "http://test-datax-sp",
5
  "password": "29c719dd-f2b3-46de-b71c-4004fb6116ee",
6
  "tenant": "42f988bf-86f1-42af-91ab-2d7cd011db42"
7
}



Set the service principal details as environment variables. During the code walk-through, you will see why these specific variables are required

Java
 




xxxxxxxxxx
1


 
1
export AZURE_CLIENT_ID="<appId>"
2
export AZURE_CLIENT_SECRET="<password>"
3
export AZURE_TENANT_ID="<tenant>"



You are all set!

Create an Azure Data Explorer Cluster

Java
 




xxxxxxxxxx
1


 
1
./goadx cluster create --name <name of the cluster you want to create> --loc <azure region for the cluster> --rg <name of the resource group> --sub <azure subscription id>



For example:

Java
 




xxxxxxxxxx
1


 
1
./goadx cluster create --name MyAdxCluster --loc "Southeast Asia" --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342



This will create a single instance cluster with the DevNoSLAStandardD11V2 VM

this is hard coded for sake of simplicity/ease-of-use

The cluster creation will take some time, and the code blocks for that time period. Please be patient ;) Grab some coffee, check Twitter or do whatever keeps you busy for about ~10-15 mins. Once the cluster is created, you will see this message:

Java
 




xxxxxxxxxx
1


 
1
created cluster MyAdxCluster with ID /subscriptions/9a42a42f-ae42-4242-b6a7-eea0ea42d342/resourceGroups/my-adx-rg/providers/Microsoft.Kusto/Clusters/MyAdxCluster and type Microsoft.Kusto/Clusters


List Down All the Clusters

You just created a cluster, let's make sure you can get its info (including any other cluster you might already have in your resource group)

Java
 




xxxxxxxxxx
1


 
1
./goadx cluster list --rg <name of the resource group> --sub <azure subscription id>



for example

Java
 




xxxxxxxxxx
1


 
1
./goadx cluster list --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342



You will get a tabular output as such:

Java
 




xxxxxxxxxx
1


 
1
+---------------+---------+----------------+-----------+-------------------------------------------------------+
2
|     NAME      |  STATE  |    LOCATION    | INSTANCES |                          URI                          |
3
+---------------+---------+----------------+-----------+-------------------------------------------------------+
4
| MyAdxCluster | Running | Southeast Asia |         1 | https://MyAdxCluster.southeastasia.kusto.windows.net |
5
+---------------+---------+----------------+-----------+-------------------------------------------------------+


You Have the Cluster, Its Time to Create a Database

To create a database in the cluster

Java
 




xxxxxxxxxx
1


 
1
./goadx db create --name <name of the database you want to create> --cluster <nname of the adx cluster> --loc <azure region> --rg <resource group name> --sub <azure subcription id>



for example

Java
 




xxxxxxxxxx
1


 
1
./goadx db create --name testadxdb --cluster MyAdxCluster --loc "Southeast Asia" --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342



It shouldnt take a too long. Once the DB is created, you will see this message:

Java
 




xxxxxxxxxx
1


 
1
created DB MyAdxCluster/testadxdb with ID /subscriptions/9a42a42f-ae42-4242-b6a7-eea0ea42d342/resourceGroups/my-adx-rg/providers/Microsoft.Kusto/Clusters/MyAdxCluster/Databases/testadxdb and type Microsoft.Kusto/Clusters/Databases


To Check the DB You Just Created...

Java
 




xxxxxxxxxx
1


 
1
./goadx db list --cluster <name of the adx cluster> --rg <resource group name> --sub <azure subscription id>



for example

Java
 




xxxxxxxxxx
1


 
1
./goadx db list --cluster MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342



You will get a tabular output as such:

Java
 




xxxxxxxxxx
1


 
1
+---------------------------+-----------+----------------+------------------------------------+
2
|           NAME            |   STATE   |    LOCATION    |                TYPE                |
3
+---------------------------+-----------+----------------+------------------------------------+
4
| MyAdxCluster/testadxdb     | Succeeded | Southeast Asia | Microsoft.Kusto/Clusters/Databases |
5
+---------------------------+-----------+----------------+------------------------------------+


You can continue experimenting further and create more clusters and/or databases

Time to Clean Up...

You can delete the cluster directly as such:

Java
 




xxxxxxxxxx
1


 
1
./goadx cluster delete --name <name of adx cluster> --rg <resource group name> --sub <azure subcription id>



example:

Java
 




xxxxxxxxxx
1


 
1
./goadx cluster delete --name MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342



You should see a confirmation message indicating that the cluster has been deleted:

Java
 




xxxxxxxxxx
1


 
1
deleted ADX cluster MyAdxCluster from resource group my-adx-rg



Or you could just delete the database itself

Java
 




xxxxxxxxxx
1


 
1
./goadx db delete --name <database name> --cluster <adx cluster name> --rg <resource group name> --sub <azure subscription id>



For example

Java
 




xxxxxxxxxx
1


 
1
./goadx db delete --name testadxdb --cluster MyAdxCluster --rg my-adx-rg --sub 9a42a42f-ae42-4242-b6a7-eea0ea42d342



You should see a confirmation message indicating that the database has been deleted:

Java
 




xxxxxxxxxx
1


 
1
deleted DB testadxdb from cluster MyAdxCluster



Alright, now that you what can be done, it's time to see how its done!

Code Walkthrough

At a high level, the application consists of two parts:

  • Azure Data Explorer operations (create, read, delete) - part of the ops package
  • CLI commands - part of the cli package

To be honest, the CLI bit is not important (at-least in the context of the blog). It uses the (ubiquitous) Cobra package to implement the cluster and cluster (top-level) commands and the sub-commands - create, list and delete. If you're interested, take a look at the following:

  • root.go - defines the root command for CLI entry point
  • cluster.go - defines and implements sub-commands for cluster operations
  • db.go - defines and implements sub-commands for database operations

The CLI part simply invokes CRUD oeprations on Azure Data Explorer resources which is the important part. So, lets go through them

Authentication

This is handled in ops/client.go. To get a handle for the client components to execute the required operations, we need to authenticate. For e.g. to get the kusto.ClustersClient:

Java
 




xxxxxxxxxx
1
10


 
1
func getClustersClient(subscription string) kusto.ClustersClient {
2
    client := kusto.NewClustersClient(subscription)
3
    authR, err := auth.NewAuthorizerFromEnvironment()
4
    if err != nil {
5
        log.Fatal(err)
6
    }
7
    client.Authorizer = authR
8
 
          
9
    return client
10
}



The part to note is the auth.NewAuthorizerFromEnvironment() which looks for a set of pre-defined environment variables - this is User environment-based authentication

Cluster Operations

ops/cluster.go is where the cluster create, list and delete is handled. Here is the snippet for create:

Java
 




xxxxxxxxxx
1


 
1
func CreateCluster(sub, clusterName, location, rgName string) {
2
    ...
3
    result, err := client.CreateOrUpdate(ctx, rgName, clusterName, kusto.Cluster{Location: &location, Sku: &kusto.AzureSku{Name: kusto.DevNoSLAStandardD11V2, Capacity: &numInstances, Tier: kusto.Basic}})
4
    err = result.WaitForCompletionRef(context.Background(), client.Client)
5
    r, err := result.Result(client)
6
    ...
7
}



CreateOrUpdate is used to initiate the cluster creation. Since this is a long running op, it returns a ClustersCreateOrUpdateFuture which we them wait on (using WaitForCompletionRef)

Note that I have used context.Background(), you can use a different context e,g one with timeout or one which can be cancelled.

Once that's one, we check the future for the Result

Listing all the clusters in a resource group is quite simple and can be done using ListByResourceGroup

Java
 




xxxxxxxxxx
1


 
1
func ListClusters(sub, rgName string) kusto.ClusterListResult {
2
    ctx := context.Background()
3
 
          
4
    result, err := getClustersClient(sub).ListByResourceGroup(ctx, rgName)
5
    if err != nil {
6
        log.Fatal(err.Error())
7
    }
8
    return result
9
}



Delete is similar to Create in the sense that it is a long running operation as well. Its the same drill - initiate the deletion (using Delete), wait for it to finish and then confirm the result

Java
 




xxxxxxxxxx
1


 
1
func DeleteCluster(sub, clusterName, rgName string) {
2
    ...
3
    result, err := client.Delete(ctx, rgName, clusterName)
4
    err = result.WaitForCompletionRef(context.Background(), client.Client)
5
    r, err := result.Result(client)
6
    ...
7
}


Database Operations

ops/db.go is where the database create, list and delete is handled. Here is the snippet for create:

Java
 




xxxxxxxxxx
1


 
1
func CreateDatabase(sub, rgName, clusterName, location, dbName string) {
2
    ...
3
    future, err := client.CreateOrUpdate(ctx, rgName, clusterName, dbName, kusto.ReadWriteDatabase{Kind: kusto.KindReadWrite, Location: &location})
4
    err = future.WaitForCompletionRef(context.Background(), client.Client)
5
 
          
6
    r, err := future.Result(client)
7
    kdb, _ := r.Value.AsReadWriteDatabase()
8
    ...
9
}



Listing is straightforward and taken care of by ListByCluster

Java
 




xxxxxxxxxx
1


 
1
func ListDBs(sub, rgName, clusterName string) kusto.DatabaseListResult {
2
    ctx := context.Background()
3
    result, err := getDBClient(sub).ListByCluster(ctx, rgName, clusterName)
4
    if err != nil {
5
        log.Fatal("failed to get databases in cluster", err)
6
    }
7
    return result
8
}



And finally, the deletion which is handled using Delete

Java
 




xxxxxxxxxx
1


 
1
func DeleteDB(sub, rgName, clusterName, dbName string) {
2
    ...
3
    future, err := getDBClient(sub).Delete(ctx, rgName, clusterName, dbName)
4
    err = future.WaitForCompletionRef(context.Background(), client.Client)
5
    r, err := future.Result(client)
6
    ...
7
}



That's it for the code walkthrough!

Conclusion

Hopefully this was helpful in demonstrating how to use the Azure Go SDK to work with Azure Data Explorer resources. You can check the documentation for Python, .NET and other examples. Happy exploring!

Topics:
Azure Data Explorer Clusters, analytics, azure, azure go sdk, big data, go, golang, tutorial

Published at DZone with permission of Abhishek Gupta , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}