Lightweight Real-Time Analytics With Spring Boot + WSO2 Siddhi

Are you looking for a way to create a lightweight real-time analytics engine? This article explains how to do exactly that by combining a fast, embeddable real-time analytics library (WSO2 Siddhi) with an immensely popular microservices platform, Spring Boot. As a bonus, I will show you how to do that in under 5 minutes with code generation! After following this article, you'll be able to generate a full-fledged Java/Spring Boot project with a documented REST API, friendly, responsive front-end, basic Spring security, comprehensive test coverage, and database integration with just a few shell commands! The generated code will be your foundation for real-time microservice application.

JHipster is a handy application generator that creates a Spring Boot and Angular application. JHipster is one of the best low code development platforms in the open source community. JHipster has become very popular on GitHub within a short amount of time. It has a high-performance Java stack on the server-side with Spring Boot and a front-end based on Angular, Bootstrap, and React. It makes project management easy with a powerful workflow, as well as build tools like Yeoman, Webpack, and Maven/Gradle. Many developers personally use it to generate multiple Spring microservices that are preconfigured to work in their company’s infrastructure. But this articles’ objective is not to give a detailed explanation of JHipster. This article will focus on how to create a JHipster module for WSO2 Siddhi. If you are new to JHipster or want to read more about JHipster, please read you can check out their documentation here: https://cbornet.github.io/.

You may have some specific real-time analytics (WSO2 Siddhi) setup that you like to use in your JHipster projects. Because you don’t want to reinvent the wheel in every project, it makes sense to abstract all the boilerplate into your own generator. In that case, you can build your own JHipster generator for WSO2 Siddhi and use it with your every JHipster project.

At Xiges.io, we heavily rely on code generation (with a custom built toolchain) to get rid of a lot of boilerplate code which we tend to program into each and every solution/project/product. When we encounter a repeatable architectural pattern we tend to apply our toolchain so, next time, we don’t have to code it again. Most of this work, including the toolchain, is part of the Xiges low code platform.

Getting Set Up

This section explains how to build the basic JHipster generator module. If you already know how to create a JHipster module, you can skip this part and go to the next section.

A JHipster module is a Yeoman generator. Also, it is an NPM package. As a first step to generate a basic yeoman generator (which would eventually be our JHipster generator) install Node.js. After that you’ll need to have Yeoman, Yarn, and Bower installed as well as the generator (yo) for creating generators. Please follow below commands with npm.

npm install -g generator-generator

npm install -g yeoman

npm install -g yo

To check whether Yeoman is installed correctly just type the yo command on the command line which will list out all the installed generators. If you can find Yeoman there, you are good to go. Finally, make sure you have git installed.

First, create a new directory in which you’ll write your JHipster generator. This directory must be named \ generator-jhipster-<name of your module(Siddhi) > . Normally, a Yeoman generator is prefixed with “generator-” but JHipster modules are prefixed with “generator-jhipster-”. Executing the below commands in order will create the folder and generate the JHipster module template.

npm install -g generator-jhipster-module

mkdir generator-jhipster-siddhi

cd generator-jhipster-siddhi

yo jhipster-module

This is not really a JHipster module that is meant to not be used in a JHipster application. This module is used to generate a new template for coding a new JHipster module for Siddhi. Essentially, it's a JHipster module to create a JHipster module. 

File structure:

As you can see, we have a readme file and package.json for the generator itself. The test folder holds tests for the generator. The index.js file is the entry point for the generator. It contains the template files for the boilerplate (for generating the actual scaffolding). We created the default generator. Now we can modify it and add in our custom features.

Scripting the Siddhi Generator

Now we will check how to customize the JHipster generator we created and add our own features in the generator with the following steps:

1. Setup and Import the generator-jhipster

The index.js file needs to export the generator-jhipster which will get run by Yeoman. Now I am going to clear everything in our generator and start from scratch. Here is what index.js file looks like after that:

const chalk = require('chalk');
const generator = require('yeoman-generator');
const packagejs = require('../../package.json');

// Stores JHipster variables
const jhipsterVar = { moduleName: 'siddhi' };

// Stores JHipster functions
const jhipsterFunc = {};

module.exports = generator.extend({

     // all your yeoman code here

});

We assign the extended generator to module.exports and make it available to the ecosystem. This is the typical method we use when we export modules in Node.js. Then we can have our own functionalities to perform through methods. For every method we added to the index.js file, the method is run once the generator is called (and usually run in sequence). Some method names have priority in this generator. The available priorities (in running order) are:

1. initializing- Initialization of your methods (getting configs and checking the project's current state).
2. prompting - prompt user preferences for the options (call this.prompt()).
3. configuring - Saving configurations and configuring the project.
4. default - If your method name doesn’t match priority and put into this group.
5. writing - copy template-files to the output folder and parsing (routes, controllers, etc).
6. conflicts - Handling conflicts in the code.
7. install - Where installations are run and add Maven dependencies to the target JHipster project pom file. (npm, Bower)
8. end - Called last, clean up.

After implementing these methods your file should be like this.

module.exports = generator.extend({

    initializing: {
        compose() {
            this.composeWith('jhipster:modules',
                { jhipsterVar, jhipsterFunc },
                this.options.testmode ? { local: require.resolve('generator-jhipster/generators/modules') } : null
            );
        },
        displayLogo() {
            // Have Yeoman greet the user.
            this.log(`Welcome to the ${chalk.bold.yellow('WSO2 siddhi')} generator! ${chalk.yellow(`v${packagejs.version}\n`)}`);
        }
    },

    prompting() {
        // return the function to call once the task is done
        const done = this.async();


        const prompts = [
            {
                type: 'input',
                name: 'userSiddhi',
                message: 'Please write your own Siddhi app',
                default: 'Do. Or do not. There is no try.'
            }
        ];

        this.prompt(prompts).then((props) => {
            this.props = props;
            // To access props later use this.props.someOption;

            done();
        });
    },

    writing() {
        // function to use directly template
        this.template = function (source, destination) {
            this.fs.copyTpl(
                this.templatePath(source),
                this.destinationPath(destination),
                this
            );
        };

        this.baseName = jhipsterVar.baseName;
        this.packageName = jhipsterVar.packageName;
        this.packageFolder = jhipsterVar.packageFolder;
        this.angularAppName = jhipsterVar.angularAppName;
        this.clientFramework = jhipsterVar.clientFramework;
        this.clientPackageManager = jhipsterVar.clientPackageManager;
        const javaDir = jhipsterVar.javaDir;
        const resourceDir = jhipsterVar.resourceDir;
        const webappDir = jhipsterVar.webappDir;



        this.template('src/main/java/package/domain/_RealtimeAnalyticsServiceImpl.java', `${javaDir}domain/RealtimeAnalyticsServiceImpl.java`);
        this.template('src/main/java/package/domain/_RealtimeAnalyticsService.java', `${javaDir}domain/RealtimeAnalyticsService.java`);
        this.template('src/main/java/package/domain/_TemperatureData.java', `${javaDir}domain/TemperatureData.java`);

        this.template('src/main/java/package/web/rest/_TemperatureDataResource .java', `${javaDir}web/rest/TemperatureDataResource.java`);
    },

    install() {


        this.addMavenDependency('org.wso2.siddhi', 'siddhi-query-api', '4.1.7');
        this.addMavenDependency('org.wso2.siddhi', 'siddhi-query-compiler', '4.1.7');
        this.addMavenDependency('org.wso2.siddhi', 'siddhi-annotations', '4.1.7');
        this.addMavenDependency('org.wso2.siddhi', 'siddhi-core', '4.1.7');
    },

    end() {
        this.log('End of WSO2 Siddhi generator');
    }
});

JHipster module uses composability, which is one of the main functions in Yeoman. The “composeWith” (this.composeWith()) method in Yeoman generators allows the generator to run parallel with another generator and it can use features from the other generator instead of having to do it all by itself. In the above example, our code composes with the “jhipster:modules” sub-generator and gets access to JHipster’s variables and functions.

As you can see in the code, we add  aJHipster function (under the install() phase) to add Maven dependencies into the pom.xml file (addMavenDependency). We also used JHipster global variables. Here are the short definitions for each variable.

baseName: the name of the application
packageName: the Java package name
angularAppName: the Angular application name
javaDir: the directory for the Java application, including the package folder
resourceDir: the directory containing the Java resources (always src/main/resources) webappDir: the directory containing the Web application (always src/main/webapp)

You can see Java Spring Boot classes and resource files add to the writing() phase and this is done by calling the template function (all the Java Spring Boot classes are described in the next section ). If you are interested in other functions available in the JHipster module, please visit https://cbornet.github.io/modules/creating_a_module.html.

2. Initializing the Generator

We initialize our generator with package.json. In the above code, Node invokes the require() function with a local file path as the function’s only argument and it gets the package.json file. This file is a Node.js module manifest. Our package.json file must contain the following:

{
  "name": "@xiges/generator-jhipster-siddhi",
  "version": "0.0.7",
  "description": "JHipster module, additional siddhi support in your JHipster application",
  "keywords": [
    "yeoman-generator",
    "jhipster-module",
    "siddhi"
  ],
  "homepage": "https://github.com/xiges/generator-jhipster-siddhi",
  "author": {
    "name": "Tharindu Vibuddha",
    "email": "tharinduvibuddha@gmail.com",
    "url": "https://github.com/xiges/generator-jhipster-siddhi"
  },
  "files": [
    "generators"
  ],
  "main": "generators/app/index.js",
  "repository": {
    "type": "git",
    "url": "git+https://github.com/xiges/generator-jhipster-siddhi"
  },
  "dependencies": {
    "yeoman-generator": "1.1.1",
    "chalk": "1.1.3",
    "mkdirp": "0.5.1",
    "generator-jhipster": ">=4.1.0"
  },
  "devDependencies": {
    "fs-extra": "0.26.4",
    "gulp": "^3.9.0",
    "gulp-bump": "1.0.0",
    "gulp-eslint": "1.0.0",
    "gulp-exclude-gitignore": "1.0.0",
    "gulp-git": "1.6.1",
    "gulp-istanbul": "0.10.3",
    "gulp-mocha": "2.2.0",
    "gulp-nsp": "2.3.0",
    "gulp-plumber": "1.0.1",
    "gulp-rename": "^1.2.0",
    "gulp-sequence": "0.4.4",
    "gulp-shell": "^0.5.1",
    "mocha": "2.3.4",
    "yeoman-assert": "2.1.1",
    "yeoman-test": "1.6.0"
  },
  "scripts": {
    "test": "mocha test/*"
  },
  "license": "Apache-2.0",
  "bugs": {
    "url": "https://github.com/xiges/generator-jhipster-siddhi/issues"
  }
}

I configured the following:

• yo: CLI tool for running Yeoman generators.
• generator-mocha: a generator for the mocha test-framework.
• gulp: a front-end build tool.
• mocha: mocha is a JavaScript test framework for Node.js programs.

3. Create a Gulp File

We use the Gulp file as our build system. Tools like Gulp are often referred to as “build tools” because they are tools for running the tasks for building a web application. Your Gulp file must be look like this.

const gulp = require('gulp');
const bumper = require('gulp-bump');
const eslint = require('gulp-eslint');
const git = require('gulp-git');
const shell = require('gulp-shell');
const fs = require('fs');
const sequence = require('gulp-sequence');
const path = require('path');
const mocha = require('gulp-mocha');
const istanbul = require('gulp-istanbul');
const nsp = require('gulp-nsp');
const plumber = require('gulp-plumber');

gulp.task('eslint', () => gulp.src(['gulpfile.js', 'generators/app/index.js', 'test/*.js'])
    // .pipe(plumber({errorHandler: handleErrors}))
        .pipe(eslint())
        .pipe(eslint.format())
        .pipe(eslint.failOnError())
);

gulp.task('nsp', (cb) => {
    nsp({ package: path.resolve('package.json') }, cb);
});

gulp.task('pre-test', () => gulp.src('generators/app/index.js')
    .pipe(istanbul({
        includeUntested: true
    }))
    .pipe(istanbul.hookRequire())
);

gulp.task('test', ['pre-test'], (cb) => {
    let mochaErr;

    gulp.src('test/*.js')
        .pipe(plumber())
        .pipe(mocha({ reporter: 'spec' }))
        .on('error', (err) => {
            mochaErr = err;
        })
        .pipe(istanbul.writeReports())
        .on('end', () => {
            cb(mochaErr);
        });
});

gulp.task('bump-patch', bump('patch'));
gulp.task('bump-minor', bump('minor'));
gulp.task('bump-major', bump('major'));

gulp.task('git-commit', () => {
    const v = `update to version ${version()}`;
    gulp.src(['./generators/**/*', './README.md', './package.json', './gulpfile.js', './.travis.yml', './travis/**/*'])
        .pipe(git.add())
        .pipe(git.commit(v));
});

gulp.task('git-push', (cb) => {
    const v = version();
    git.push('origin', 'master', (err) => {
        if (err) return cb(err);
        git.tag(v, v, (err) => {
            if (err) return cb(err);
            git.push('origin', 'master', {
                args: '--tags'
            }, cb);
            return true;
        });
        return true;
    });
});

gulp.task('npm', shell.task([
    'npm publish'
]));

function bump(level) {
    return function () {
        return gulp.src(['./package.json'])
            .pipe(bumper({
                type: level
            }))
            .pipe(gulp.dest('./'));
    };
}

function version() {
    return JSON.parse(fs.readFileSync('package.json', 'utf8')).version;
}

gulp.task('prepublish', ['nsp']);
gulp.task('default', ['static', 'test']);
gulp.task('deploy-patch', sequence('test', 'bump-patch', 'git-commit', 'git-push', 'npm'));
gulp.task('deploy-minor', sequence('test', 'bump-minor', 'git-commit', 'git-push', 'npm'));
gulp.task('deploy-major', sequence('test', 'bump-major', 'git-commit', 'git-push', 'npm'));

It’s often used to do front-end tasks like:

• Spinning up a web server.
• Reloading the browser automatically whenever a file is saved.
• Using preprocessors like Sass or LESS.
• Optimizing assets like CSS, JavaScript, and images.

So now we have finished creating a basic JHipster generator for WSO2 Siddhi. But now we have to configure Spring Boot with real-time analytics (WSO2 Siddhi).

Spring Boot + Real-Time Analytics

Spring Boot is a very popular Java-based framework for building web and enterprise-based applications. Spring framework provides a wide variety of features addressing modern business needs. As their docs taut, "Spring Boot makes it easy to create stand-alone, production-grade Spring-based applications that can 'just run.'" Spring Boot takes an opinionated view of the Spring platform and combines third-party libraries so developers can start with minimum fuss. Most Spring Boot applications need only a little bit of Spring configuration, so it’s easy to embed WSO2 Siddhi to Spring Boot. Let's see how it’s done.

Before we go any further, we have to understand the main concepts behind real-time analytics with WSO2 Siddhi. WSO2 Siddhi is a Java library which carries out real-time processing on complex events. The streaming SQL Language of Siddhi is being used to describe complex conditions from the data streams. Siddhi is able to perform both Stream and complex event processing. The below diagram shows the basic workflow of the WSO2 Siddhi 3.0.

Now we are embedding Siddhi in a Java Spring Boot project, allowing us to use the Siddhi query language to carry out real-time processing of complex events without running a WSO2 CEP server.

Step 1: Implementing Business Service and Adding POST Rest Service for Siddhi Application

As a first step, we need to define a stream definition and Siddhi query. Stream definitions always define the format of your incoming events and the query is defined as below:

String definition = "define stream TempStream(roomNo int, temperature double,
deviceId long);"

String query = "@info(name = 'avgTemperature') " +
"from TempStream#window.time(60 sec) " +
"select avg(temperature) as temperature,deviceID " +"group by roomNo " +
"insert into AvgTempStream ;";

This Siddhi query stores incoming events for 60 econds, groups them by roomNo and calculates the average temperature. Then it inserts the results into a stream named AvgTempStream.

Step 2: Creating a Siddhi Runtime

This step involves creating a runtime representation of a siddhiAppRuntime by combining the stream definition and the Siddhi query you created in Step 1.

SiddhiManager siddhiManager = new SiddhiManager();
//Generating runtime
SiddhiAppRuntime siddhiAppRuntime = siddhiManager
.createSiddhiAppRuntime(definition+query);

The Siddhi Manager parses the Siddhi App and provides you with a Siddhi app runtime. This Siddhi app runtime is used to add callbacks and input handlers to the Siddhi app runtime.

Step 3: Registering a Callback

Siddhi has two types of callbacks:

• Stream Callback — this subscribes to an event stream.
• Query Callback — this subscribes to a query.

We need a callback to retrieve output events from the query. So we can register a callback to the Siddhi app runtime. When results are generated, they are sent to the receive method of this callback. Also, we can print the incoming events from an event printer which is added inside this callback.

For example:

siddhiAppRuntime.addCallback("AvgTempStream", new QueryCallback() {
            @Override
            public void receive(Event[] events) {
                EventPrinter.print(events);
            }
        });

Step 4: Sending Events

As a final step, you need to send events from the event stream to the query and you need to obtain an input handler as shown below:

//Retrieving input handler to push events into Siddhi        
InputHandler inputHandler =siddhiAppRuntime .getInputHandler("StockEventStream");         
//Starting event processing        
siddhiAppRuntime.start();         
//Sending events to Siddhi        
inputHandler.send(new Object[]{2, 23.0, 100L});

Refer to these files at the bottom of the article for the exact implementation of the Business Service and adding a POST Rest Service to our Siddhi application.

@Service
public class RealtimeAnalyticsServiceImpl extends StreamCallback implements
        RealtimeAnalyticsService,ApplicationListener<ContextRefreshedEvent> {

    private  static final Logger log = LoggerFactory.getLogger(RealtimeAnalyticsServiceImpl.class);
    private SiddhiManager siddhiManager;
    private SiddhiAppRuntime siddhiAppRuntime;
    private InputHandler temperatureInputHandler;
    private static final String script = "define stream TempStream(roomNo int, temperature double, deviceId long); " +
            " " +
            "@info(name = 'avgTemperature') " +
            "from TempStream#window.time(60 sec)  " +
            "select avg(temperature) as temperature,deviceId " +
            "group by roomNo " +
            "insert into AvgTempStream ;";

    @Override
    public void onApplicationEvent(ContextRefreshedEvent contextRefreshedEvent) {

        log.info("Realtime analytics engine starting up");
        siddhiManager = new SiddhiManager();
        Map<String,String> stockConfig = new HashMap<>();

        InMemoryConfigManager inMemoryConfigManager = new InMemoryConfigManager(stockConfig,Collections.emptyMap());
        siddhiManager.setConfigManager(inMemoryConfigManager);
        // siddhiManager.setExtension();
        siddhiAppRuntime = siddhiManager.createSiddhiAppRuntime(script);

        siddhiAppRuntime.addCallback("AvgTempStream",this);

        temperatureInputHandler = siddhiAppRuntime.getInputHandler("TempStream");

        siddhiAppRuntime.start();

    }



    @PreDestroy
    private void onApplicationExit(){

        log.info("Shutting down  WSO2 Siddhi");
        siddhiAppRuntime.shutdown();
        siddhiManager.shutdown();
    }

    @Override
    public void inboundDataEvent(String tempId,  TemperatureData data) {
        try {
            temperatureInputHandler.send(new Object[]{data.getRoomNo(),data.getTemperature(),data.getDeviceId()});
        } catch (InterruptedException e) {
            log.error("error occurred while feeding temperature data ", e);
        }
    }



    @Override
    public void receive(Event[] events) {



        // EventPrinter.print(events);
        log.debug("Avg Temperature",events);

    }
)
}

public interface RealtimeAnalyticsService {

    void inboundDataEvent (String temperatureId , TemperatureData data);
}

public class TemperatureData {
    private int roomNo;
    private double temperature;
    private Long deviceId;


    public int getRoomNo() {
        return  roomNo;
    }

    public void setRoomNo(int roomNo) {
        this.roomNo = roomNo;
    }

    public double getTemperature (){
        return temperature;
    }

    public void setTemperature(double temperature) {
        this.temperature = temperature;
    }


    public Long getDeviceId() {
        return deviceId;
    }

    public void setDeviceID(Long deviceId) {
        this.deviceId = deviceId;
    }


    @Override
    public String toString(){
        return "TemperatureData{" +
                "roomNo='" + roomNo + '\'' +
                ", temperature=" +temperature +
                ", deviceId='" + deviceId+ '\'' +
                '}';
    }


}

@RestController
public class TemperatureDataResource {
    private final Logger log = LoggerFactory.getLogger(TemperatureDataResource.class);
    @Autowired
    RealtimeAnalyticsService realtimeAnalyticsService;

    @PostMapping("/temperature-data")

    public ResponseEntity<Void> addTemperatureData(@PathVariable String tempId, @RequestBody TemperatureData data){

        TemperatureData temperatureData = new  TemperatureData();
        temperatureData.setRoomNo(data.getRoomNo());
        temperatureData.setDeviceID(data.getDeviceID());
        temperatureData.setTemperature(data.getTemperature());
        realtimeAnalyticsService.inboundDataEvent(tempId,data);
        return new ResponseEntity<Void>(HttpStatus.OK);
    }

}

Conclusion

Creating a JHipster module with WSO2 Siddhi is an easy way to simplify your microservice generation, especially if your microservice uses the same configuration. Since it is a module, it’s very easy to add functionalities and meet your needs. Our company uses this real-time analytics for making real-time predictions and tracking with our IoT products.

Below is the GitHub repository with the module used in this blog. Feel free to fork it and make changes to match your company requirements!

Now you can download JHipster generator for Siddhi by using this command

npm install g generator-jhipster-siddhi

Then run the module on a JHipster generated application.

yo jhipster-siddhi

Soon this module will be available in the JHipster marketplace (after the JHipster team verifies it).

Link for the source code - https://github.com/xiges/generator-jhipster-siddhi.

Link for the npm registry - https://www.npmjs.com/package/generator-jhipster-siddhi.

If you have any suggestions or enhancements you want to see in the module, please create an issue in the Xiges GitHub repository — https://github.com/xiges/generator-jhipster-siddhi/issues.

That’s all I have for this topic. Thanks for reading. Until next time, happy coding!

References

• https://yeoman.io/authoring/composability.html
• https://www.jhipster.tech/modules/creating-a-module
• https://www.baeldung.com/jhipster
• https://yeoman.io/authoring/file-system.html
• https://gulpjs.com/docs/en/getting-started/javascript-and-gulpfiles
• https://siddhi-io.github.io/siddhi/#try-siddhi-with-wso2-stream-processor
• https://docs.wso2.com/display/CEP400/SiddhiQL+Guide+3.0