{{announcement.body}}
{{announcement.title}}

Distributed SQL Change Management With Liquibase and YugabyteDB on GKE

DZone 's Guide to

Distributed SQL Change Management With Liquibase and YugabyteDB on GKE

In this article, look at database SQL change management with Liquibase and Yugabyte on GKE.

· Database Zone ·
Free Resource

Liquibase is an open source and extensible change management project that supports a variety of databases including Snowflake, MySQL, and PostgreSQL via JDBC. Liquibase allows users to easily define changes in SQL, XML, JSON, and YAML. These changes are then managed in a version control system so the changes can be documented, ordered, and standardized. For more information on the features and benefits of Liquibase, check out their documentation site.

In this blog post we’ll show you how to:

  • Install a 3 node YugabyteDB cluster on Google Kubernetes Engine
  • Build the sample Northwind database
  • Install and configure Liquibase
  • Create a simple changeset and verify the results
  • Explore how changes are documented and managed in Liquibase

New to distributed SQL or YugabyteDB? Read on.

What Is Distributed SQL?

Distributed SQL databases are becoming popular with organizations interested in moving data infrastructure to the cloud or cloud native environments. This is often motivated by the desire to reduce TCO or move away from the horizontal scaling limitations of monolithic RDBMS like Oracle, PostgreSQL, MySQL, and SQL Server. The basic characteristics of Distributed SQL are:

  • They must have a SQL API for querying and modeling data, with support for traditional RDBMS features like foreign keys, partial indexes, stored procedures, and triggers.
  • Automatic distributed query execution so that no single node becomes a bottleneck.
  • Should support automatic and transparent distributed data storage. This includes indexes, which should be sharded across multiple nodes of the cluster so that no single node becomes a bottleneck. Data distribution ensures high performance and high availability.
  • Distributed SQL systems should also provide for strongly consistent replication and distributed ACID transactions.

For a deeper discussion about what Distributed SQL is, check out, “What is Distributed SQL?”

What Is YugabyteDB?

YugabyteDB is an open source, high-performance distributed SQL database built on a scalable and fault-tolerant design inspired by Google Spanner. YugabyteDB is PostgreSQL wire compatible, cloud native, offers deep integration with GraphQL projects, plus supports advanced RDBMS features like stored procedures, triggers, and UDFs.

Got questions? Make sure to ask them in our YugabyteDB Slack channel. Ok, let’s dive in…

Step 1: Install YugabyteDB on a GKE Cluster Using Helm 3

In this section we are going to install YugabyteDB on the cluster. The complete steps are documented here. We’ll assume you already have a GKE cluster up and running as a starting point.

The first thing to do is to add the charts repository.

Java
 




x


 
1
$ helm repo add yugabytedb https://charts.yugabyte.com



Now, fetch the updates.

Java
 




xxxxxxxxxx
1


 
1
$ helm repo update



Create a namespace. In this case we’ll call it yb-demo.

Java
 




xxxxxxxxxx
1


 
1
$ kubectl create namespace yb-demo



Expected output:

Java
 




xxxxxxxxxx
1


 
1
namespace/yb-demo created



We are now ready to install YugabyteDB. In the command below we’ll be specifying values for a resource constrained environment.

Java
 




xxxxxxxxxx
1


 
1
$ helm install yb-demo yugabytedb/yugabyte \
2
--set resource.master.requests.cpu=1,resource.master.requests.memory=1Gi,\
3
resource.tserver.requests.cpu=1,resource.tserver.requests.memory=1Gi,\
4
enableLoadBalancer=True --namespace yb-demo --wait



To check the status of the cluster, execute the below command:

Java
 




xxxxxxxxxx
1


 
1
$ kubectl get services --namespace yb-demo



Note the external-IP for yb-tserver-service which we are going to use to establish a connection between YugabyteDB and Liquibase. From the screenshot above we can see that the IP is 34.72.XX.XX and the YSQL port is 5433.

Step 2: Creating the Northwind Sample Database

The next step is to download a sample schema and data. You can find a variety of sample databases that are compatible with YugabyteDB in our Docs. For the purposes of this tutorial we are going to use the Northwind sample database. The Northwind database contains the sales data for a fictitious company called “Northwind Traders,” which imports and exports specialty foods from around the world. The Northwind database is an excellent tutorial schema for a small-business ERP, with customers, orders, inventory, purchasing, suppliers, shipping, employees, and single-entry accounting.

Connect to the yb-tserver-pod by running the following command:

Java
 




xxxxxxxxxx
1


 
1
$ kubectl exec -n yb-demo -it yb-tserver-0 /bin/bash



To download the schema and data files, run the following commands:

Java
 




xxxxxxxxxx
1


 
1
$ wget https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/northwind_ddl.sql
2
$ wget https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/northwind_data.sql



Note: If the Google Cloud Shell tells you that the wget command does not exist, you can execute:

Java
 




xxxxxxxxxx
1


 
1
$ yum install wget -y



To connect to the YSQL service exit out of the pod shell and run the following command:

Java
 




xxxxxxxxxx
1


 
1
$ exit 
2
$ kubectl exec -n yb-demo -it yb-tserver-0 -- ysqlsh -h yb-tserver-0.yb-tservers.yb-demo



Create a database and connect to it using the following commands:

Java
 




xxxxxxxxxx
1


 
1
yugabyte=# CREATE DATABASE northwind;
2
northwind=# \c northwind;



We can now create the database objects and load them with data using the files we downloaded to yb-tserver-pod using the following commands:

Java
 




xxxxxxxxxx
1


 
1
northwind=# \i 'northwind_ddl.sql';
2
northwind=# \i 'northwind_data.sql';



Verify that the tables are created by running the following command:

Java
 




xxxxxxxxxx
1
20


 
1
northwind-# \d
2
 
          
3
                 List of relations
4
 Schema |          Name          | Type  |  Owner
5
--------+------------------------+-------+----------
6
 public | categories             | table | yugabyte
7
 public | customer_customer_demo | table | yugabyte
8
 public | customer_demographics  | table | yugabyte
9
 public | customers              | table | yugabyte
10
 public | employee_territories   | table | yugabyte
11
 public | employees              | table | yugabyte
12
 public | order_details          | table | yugabyte
13
 public | orders                 | table | yugabyte
14
 public | products               | table | yugabyte
15
 public | region                 | table | yugabyte
16
 public | shippers               | table | yugabyte
17
 public | suppliers              | table | yugabyte
18
 public | territories            | table | yugabyte
19
 public | us_states              | table | yugabyte
20
(14 rows)



Verify we have data by issuing a simple SELECT:

Java
 




xxxxxxxxxx
1


 
1
northwind=# SELECT count(*) FROM products;
2
 count
3
-------
4
    77
5
(1 row)



By default, a YugabyteDB installation doesn’t have a password setup for the default yugabyte user. Specifying one is done the same way you’d do it in PostgreSQL.

Java
 




xxxxxxxxxx
1


 
1
northwind=# ALTER ROLE yugabyte WITH PASSWORD 'password';



Step 3: Install and Configure Liquibase

Locate the appropriate version of Liquibase for your platform from their downloads page.

https://www.liquibase.org/download

For the purposes of this demo, we’ll be installing Liquibase locally on a Mac.

Create a Liquibase Project

On my Mac I created a new folder called LiquibaseYugabyteDB.

Download the Appropriate PostgreSQL Driver

Visit https://jdbc.postgresql.org/download.html and download the appropriate JDBC driver for your environment. I downloaded JDBC 4.2 (42.2.14) and placed it in the LiquidbaseYugabyteDB folder.

Create a Changelog File

Next, in this same folder I created a file called dbchangelog.xml file. This changelog contains the sequence of changesets, each one of which makes small changes to the structure of the database. Add the following boilerplate to this file to get started.

Java
 




xxxxxxxxxx
1


 
1
<?xml version="1.0" encoding="UTF-8"?>  
2
  <databaseChangeLog  
3
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"  
4
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
5
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
6
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">    
7
  </databaseChangeLog>


Create a Properties File

Now in the same directory, create a file called liquibase.properties and add the following content to it that is reflective of your environment.

Java
 




xxxxxxxxxx
1


 
1
changeLogFile:  ../LiquibaseYugabyteDB/dbchangelog.xml  
2
    url:  jdbc:postgresql://34.72.XX.XX:5433/northwind
3
    username:  yugabyte  
4
    password:  password
5
    driver:  org.postgresql.Driver  
6
    classpath:  ../LiquibaseYugabyteDB/postgresql-42.2.14.jar


Step 4: Create a Changeset and Verify the Results

Create a Changeset

To create a changeset, return to the dbchangelog.xml file and add the following copy highlighted in bold. In the changeset we are going to create a promotions table in the Northwind database with columns for the name of the promotion and the discounted amount.

Java
 




xxxxxxxxxx
1
19


 
1
<?xml version="1.0" encoding="UTF-8"?>  
2
  <databaseChangeLog  
3
    xmlns="http://www.liquibase.org/xml/ns/dbchangelog"  
4
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  
5
    xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
6
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd">  
7
 
          
8
    <changeSet  id="1"  author="jguerrero">  
9
        <createTable  tableName="promotions">  
10
            <column  name="id"  type="serial">  
11
                <constraints  primaryKey="true"  nullable="false"/>  
12
            </column>  
13
            <column  name="name"  type="varchar(50)">  
14
                <constraints  nullable="false"/>  
15
            </column>  
16
            <column  name="discount"  type="numeric(3,2)"/>                  
17
        </createTable>  
18
   </changeSet>
19
  </databaseChangeLog>



The SQL equivalent of the changeset above would be:

Java
 




xxxxxxxxxx
1


 
1
CREATE  TABLE  promotions  
2
  (id serial PRIMARY KEY,  
3
   name  VARCHAR(50) UNIQUE NULL,  
4
   discount  NUMERIC (3,2) 
5
  );



Execute the Changeset

Execute the update command locally in the directory you created to send the changeset to the database.

Java
 




xxxxxxxxxx
1


 
1
$ liquibase update
2
Liquibase Community 4.0.0 by Datical
3
Starting Liquibase at 11:00:28 (version 4.0.0 #19 built at 2020-07-13 19:45+0000)
4
Liquibase: Update has been successful.



Verify the Results

To verify that the promotions table was created we can open up a YSQL shell and insert two records into the table and select the data out.

Java
 




xxxxxxxxxx
1


 
1
INSERT INTO promotions 
2
(id, name, discount)
3
VALUES 
4
(DEFAULT, 'Memorial Day Promotion', 3.50),
5
(DEFAULT, 'International Coffee Day Promotion', 5.35);
6
 
          
7
SELECT * FROM promotions;


Step 5: Managing Changes

Along with the promotions table, you should now see two additional tables that Liquibase has created in the Northwind database.

The databasechangelog Table

This table keeps a record of all the changesets that were deployed. This means that the next time you deploy again, the changesets in the changelog will be compared with the databasechangelog tracking table and only the new changesets that were not found in the databasechangelog will be deployed. You will notice that a new row was created in that table with the changeset information we have just deployed. For this example:

The databasechangelock Table

This table is used internally by Liquibase to manage access to the changelog table during deployment. So, nothing to see here!

Step 6: Issue a Second Changeset

Create a Second Changeset

Let’s create an additional change set. In this case let’s add two more columns to the promotions table. A start_date column and a stop_date column so we know when a promotion starts and ends. Instead of XML, let’s use SQL to initiate the changes. To do this, create a file in your Liquibase project directory called changelog.sql.

Add the following commands to this file:

Java
 




xxxxxxxxxx
1


 
1
--liquibase formatted sql  
2
--changeset jimmy:1
3
ALTER TABLE promotions
4
ADD COLUMN start_date TIMESTAMP,
5
ADD COLUMN stop_date TIMESTAMP;



Issue the following command to send the change to the database.

Java
 




xxxxxxxxxx
1


 
1
$ liquibase --changeLogFile=changelog.sql update



Verify the Change

We should now be able to see a second record in the databasechangelog tracking table.

We should also be able to see the new start_time and stop_date columns in the promotions table.

Conclusion

That’s it! You now have a PostgreSQL-compatible, 3 node YugabyteDB cluster running on GKE whose changes are now being managed and tracked by Liquibase. To learn more about supported third-party PostgreSQL tools that work with YugabyteDB, check out the YugabyteDB Documentation or join our community Slack and hit us up there.

Topics:
database, distributed sql, distributed sql databases, kubernetes, postgres, postgresql, yugabyte

Published at DZone with permission of Jimmy Guerrero . See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}