NoSQL Support for Hyperlambda

Handling huge amounts of data is literally impossible without having some sort of relationship to NoSQL database systems, such as Cassandra or ScyllaDB. The reasons are because these database systems are created to scale linearly, implying they're just as fast with a billion records as they are with 50 records. Hence, building things such as Twitter or Facebook is impossible without the ability to use NoSQL.

Hyperlambda has always suffered from lack of official support for such NoSQL database systems. However, yesterday that change with the release of Magic version 10.0.19. Out of the box you can now as of the latest release of Magic and Hyperlambda connect to any Scylla or Cassandra based cluster, retrieve records, and do any of the main CRUD operations towards your data storage. And as an additional kicker you can exchange the default RDBMS based logger with a NoSQL equivalent.

The thing I am personally most jazzed about though, is the ability to exchange the file system with a "virtual file system". This is because it allows you to create and maintain 100% perfectly stateless Magic instances, put these into containers, and use Kubernetes to load balance between hundreds of Magic instances if you wish - Yet still have these behave as one single instance. However, if you want to try this route, you'll have to first create a "files" table. Below is an example of how to do this in Cassandra and ScyllaDB.

create keyspace if not exists magic
   with replication = {
      'class': 'NetworkTopologyStrategy',
      'replication_factor': 5
   };

use magic;

create table if not exists files(
   tenant text,
   cloudlet text,
   folder text,
   filename text,
   content blob,
   primary key((tenant, cloudlet), folder, filename));

When you have created your table you can insert whatever files you've got locally by using the following snippet of Hyperlambda. Notice, you'll have to modify your "tenant" and "cloudlet" parts according to how your files are stored locally on your own machine. Use the "Eval" menu item in Magic to execute the following Hyperlambda.

/*
 * Inserts all dynamic files and folders into the magic CQL database.
 */
cql.connect:[generic|magic]

   /*
    * The root folder where your Magic backend is running.
    */
   .tenant:Users
   .cloudlet:"thomashansen/Documents/projects/magic/magic/backend"

   /*
    * Inserting root folder.
    */
   cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, '/files/', '')"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet

   /*
    * Inserting appsettings.json and its folder.
    */
   config.load
   convert:x:-
      type:bytes
   cql.execute:"insert into files (tenant, cloudlet, folder, filename, content) values (:tenant, :cloudlet, '/config/', 'appsettings.json', :config)"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet
      config:x:@convert
   cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, '/config/', '')"
      tenant:x:@.tenant
      cloudlet:x:@.cloudlet

   /*
    * Inserting folders.
    */
   io.folder.list-recursively:/
      display-hidden:true
   for-each:x:-/*

      strings.concat
         .:/files
         get-value:x:@.dp/#
      
      cql.execute:"insert into files (tenant, cloudlet, folder, filename) values (:tenant, :cloudlet, :folder, '')"
         tenant:x:@.tenant
         cloudlet:x:@.cloudlet
         folder:x:@strings.concat

   /*
    * Inserting files.
    */
   io.file.list-recursively:/
      display-hidden:true
   for-each:x:-/*

      io.file.load.binary:x:@.dp/#
   
      strings.split:x:@.dp/#
         .:/
      unwrap:x:+
      .filename:x:@strings.split/0/-
      remove-nodes:x:@strings.split/0/-
      strings.join:x:@strings.split/*
         .:/
      strings.concat
         .:/files/
         get-value:x:@strings.join
         .:/
      strings.replace:x:-
         .://
         .:/
      cql.execute:"insert into files (tenant, cloudlet, folder, filename, content) values (:tenant, :cloudlet, :folder, :filename, :content)"
         tenant:x:@.tenant
         cloudlet:x:@.cloudlet
         folder:x:@strings.replace
         filename:x:@.filename
         content:x:@io.file.load.binary

remove-nodes:x:../**/io.folder.list-recursively/*
remove-nodes:x:../**/io.file.list-recursively/*

What the above does is to "slurp up" all dynamic files you've got in your Magic installation and insert these into your "files" table. If you're using the default Docker images we publish for Magic, and you want to insert the files into a table such that they will be correctly resolved from your Docker container, you can set "tenant" to "magic" and "cloudlet" to "files". This allows you to create hundreds of Magic instances, load balance between these using Kubernetes, and have they all resolve towards the same "virtual file system".

When you have done the above, you'll have to reconfigure Magic to use your virtual file system, and restart your container. The configuration settings for doing this is as follows.

{
  "magic": {
    "io": {
      "file-service": "magic.data.cql.io.CqlFileService",
      "folder-service": "magic.data.cql.io.CqlFolderService",
      "stream-service": "magic.data.cql.io.CqlStreamService"
    }
  }
}

At this point you can restart Magic's container, and it will automagically use the files inside of your Scylla or Cassandra database. To create a table to use for logging you can use the following CQL.

create keyspace if not exists magic with
   replication = {
      'class': 'NetworkTopologyStrategy',
      'replication_factor': 5
   };

use magic;

create table if not exists log_entries(
   tenant text,
   cloudlet text,
   created timeuuid,
   type text,
   content text,
   exception text,
   primary key((tenant, cloudlet), created)) with clustering order by (created desc);

alter table log_entries with default_time_to_live = 604800;

Notice the time to live setting. This will ensure that log items are automatically deleted after 7 days, which is the equivalent of 604,800 seconds. Change this as you see fit. When you have created the above table you can configure Magic to use your CQL log implementation with the following addition to your "appsettings.json" file.

{
  "magic": {
    "logging": {
      "service": "magic.data.cql.logging.Logger"
    }
  }
}

Then restart your container, and Magic will automatically use your NoSQL database for logging. Notice both of the above changes requires you to allow your Docker containers to actually access your NoSQL database. Hence, you'll have to modify Magic's "docker-compose.yml" file to somehow access your NoSQL database, either by using the host network, or by exposing the relevant ports. This is true for both your NoSQL docker images if you're using Docker to create ScyllaDB or Cassandra instances, and your Magic backend Docker image. How to do this varies according to how you intend to run things, but you should be able to figure out things with the information specified here.