LiquiCouch: A Liquibase-Like Framework to Automate Schema Changes

A lot has already been said about all the advantages of schema-less databases. They make a bunch of cumbersome tasks simpler and sometimes, much faster. But any relevant system cannot be totally "schema-less"; they expect at least some core structure to work with.

But is it really necessary to "validate" your schema from both your application and database? Sometimes, it sounds like overkill to me as most of the side effects of schema-less databases can be easily handled in the application code itself or in the query language (N1QL in our case). Of course, as your model evolves, it might get a little bit trickier to handle every single possibility since the project start.

In those cases, we commonly run data migrations (aka: execute Updates, Inserts or deletes) to migrate the "old schema" to the new one. Ideally, those migrations need to be executed automatically, sequentially, and if something goes wrong, the whole application start should fail.

Well, turns out that it is exactly what LiquiCouch does! It is a liquibase-like Java implementation for Couchbase. Let's see how it works:

With Maven:

<dependency>
  <groupId>com.github.deniswsrosa</groupId>
  <artifactId>liquicouch</artifactId>
  <version>0.6.4</version>
</dependency>

With Gradle:

compile 'org.javassist:javassist:3.18.2-GA' // workaround for ${javassist.version} placeholder issue*
compile 'com.github.liquicouch:liquicouch:0.6.4'

Usage With Spring

You need to instantiate the Liquicouch object and provide some configuration. If you use Spring, it can be instantiated as a singleton bean in the Spring context. In this case, the migration process will be executed automatically upon startup.

@Autowired
private ApplicationContext context;

@Bean
public LiquiCouch liquicouch(){
  LiquiCouch runner = new LiquiCouch(context); //It will grab all the data needed from the application.properties file
  runner.setChangeLogsScanPackage(
       "com.example.yourapp.changelogs"); // the package to be scanned for changesets

  return runner;
}

Usage Without Spring

Using LiquiCouch without a spring context has a similar configuration but you have to remember to run execute() method to start the migration process.

LiquiCouch runner = new LiquiCouch("couchbase://SOME_IP_ADDRESS", "yourBucketName", "bucketPasword");
runner.setChangeLogsScanPackage(
     "com.example.yourapp.changelogs"); // package to scan for changesets

runner.execute();         //  ------> starts migration changesets

The examples above provide minimal configuration. However, the Liquicouch object provides some other possibilities (setters) to make the tool more flexible:

runner.setEnabled(shouldBeEnabled);              // default is true, migration won't start if set to false

Creating Change Logs

ChangeLog contains a bunch of ChangeSets. ChangeSet is a single task (set of instructions made on a database). In other words, ChangeLog is a class annotated with @ChangeLog and contains methods annotated with @ChangeSet.

package com.example.yourapp.changelogs;

@ChangeLog
public class DatabaseChangelog {

  @ChangeSet(order = "1", id = "someChangeId", author = "testAuthor")
  public void importantWorkToDo(){
     // task implementation
  }

}

@ChangeLog

A class with change sets must be annotated by @ChangeLog. There can be more than one change log class but in that case, an order argument must be provided:

@ChangeLog(order = "001")
public class DatabaseChangelog {
  //...
}

ChangeLogs are sorted alphabetically (that is why it is a good practice to start the order with zeros) by order argument, and change sets are applied due to this order.

@ChangeSet

A method annotated by @ChangeSet is taken and applied to the database. The history of applied change sets is stored in a dbChangeLog type document.

Annotation Parameters:

• order — a string for sorting change sets in one changelog. It sorts alphabetically in an ascending order. It can be a number, a date etc.
• ID — a name of a change set that must be unique for all change logs in a database
• author — the author of a changeSet
• runAlways — [optional, default: false] changeset will always be executed but only the first execution event will be stored as a document.
• recounts — [optional, default: 0] [Only applied when changSet returns a ParameterizedN1qlQuery] if you want to be sure that all documents have been updated, you can return a ParameterizedN1qlQuery. This query expects a result called size. If size is not zero, the query will be executed again according to the number of recounts specified. If none of the recounts returns zero, an exception will be thrown, and the application will fail to start.
• retries — [optional, default: 0] [Only applied when changSet returns a ParameterizedN1qlQuery] if the recount operation fails (the count result isn't zero), it will rerun the changeSet in an attempt to update the remaining documents (Your changeSet should be able to run multiple times without any side effects). If all retries fail, an exception will the thrown and the application will fail to start.

Defining ChangeSet Methods

Methods annotated by @ChangeSet can have one of the following definition:

/**
 * If you are using Spring, you can Autowire your Services or Repositories
 */
@Component
@ChangeLog(order = "001")
public class Migration1 {

    @Autowired // Yes, You can a
    private UserService userService;

    @Autowired
    private UserRepository userRepository;

    @ChangeSet(order = "001", id = "someChangeId1", author = "testAuthor")
    public void importantWorkToDo(Bucket bucket){
        System.out.println("----------Migration1 - Method1");
    }

    @ChangeSet(order = "002", id = "someChangeId2", author = "testAuthor")
    public void method2(Bucket bucket){
        System.out.println("----------Migration1 - Method2");
    }

    @ChangeSet(order = "003", id = "someChangeId3", author = "testAuthor")
    public void method3(Bucket bucket){
        System.out.println("----------Migration1 - Method3");
    }

    @ChangeSet(order = "004", id = "someChangeId4", author = "testAuthor")
    public void method4(Bucket bucket){
        System.out.println("----------Migration1 - Method4");
    }

    @ChangeSet(order = "005", id = "someChangeId5", author = "testAuthor")
    public void method5(){
        System.out.println("----------Migration1 - Method5 (The bucket parameter is not necessary here)");
    }


    /**
     * Here is an example of how you can check if your update has run successfully, all you need to do is to
     * return a ParameterizedN1qlQuery.
     * @return
     */
    @ChangeSet(order = "006", id = "someChangeId6", author = "testAuthor", recounts = "2", retries = "1")
    public ParameterizedN1qlQuery method6(){

        //adding some data as an example
        userService.save(new User("someUserIdForTesting", "user1", new Address(), new ArrayList<>(), Arrays.asList("admin", "manager")));
        Iterable<User> users = userRepository.findAll(new PageRequest(0, 100));//we just care about the first 100 records

        users.forEach( e-> {
            //rename admin to adm
             if(e.getSecurityRoles().contains("admin")) {
                 e.getSecurityRoles().remove("admin");
                 e.getSecurityRoles().add("adm");
             }
            userRepository.save(e);
        });

        //IMPORTANT: The query MUST have an attribute called *size*
        String queryString = "Select count(userRole)  as size from test t unnest t.securityRoles as userRole " +
                " where t._class='com.cb.springdata.sample.entities.User' " +
                " and userRole = 'admin'";
        N1qlParams params = N1qlParams.build().consistency(ScanConsistency.REQUEST_PLUS).adhoc(true);
        ParameterizedN1qlQuery query = N1qlQuery.parameterized(queryString, JsonObject.create(), params);
        return query;
    }

}

Without Spring

/**
 * This is an example of how to use it without Spring, in this case you can execute all the queries via the Bucket argument.
 */
@ChangeLog(order = "2")
public class Migration2 {

    @ChangeSet(order = "1", id = "someChangeId21", author = "testAuthor")
    public void importantWorkToDo(){
        System.out.println("----------Migration2 - Method1");
    }

    @ChangeSet(order = "2", id = "someChangeId22", author = "testAuthor")
    public void method2(){
        System.out.println("----------Migration2 - Method2");
    }

    @ChangeSet(order = "3", id = "someChangeId23", author = "testAuthor")
    public void method3(){
        System.out.println("----------Migration2 - Method3");
    }

    @ChangeSet(order = "4", id = "someChangeId24", author = "testAuthor")
    public void method4(){
        System.out.println("----------Migration2 - Method4");
    }

    @ChangeSet(order = "5", id = "someChangeId25", author = "testAuthor")
    public void method5(Bucket bucket){
        System.out.println("----------Migration2 - Method5");
    }

    @ChangeSet(order = "6", id = "someChangeId256", author = "testAuthor", runAlways=true)
    public void method6(Bucket bucket){
        System.out.println("----------Migration2 - Method6 - THIS SHOULD ALWAYS RUN "+bucket.name());
    }
}

Demo Project

You can clone the sample project here https://github.com/deniswsrosa/liquicouch-demo

Documentation

https://github.com/deniswsrosa/liquicouch

Support

This is a community project mainly supported by me. If you have any issues or questions, just ping me at @deniswsrosa