Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Akka Persistence With Apache Ignite

DZone's Guide to

Akka Persistence With Apache Ignite

This new project aims to reap the benefits of both event sourcing and the data grid based on the Journal Plugin and TCK specs provided by Akka persistence.

· Big Data Zone ·
Free Resource

Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.

In this post, we will share a new project to use Apache Ignite data grid as an event and snapshot store to reap the benefits of both event sourcing and the data grid.

The implementation is based on the Journal Plugin and TCK specs provided by Akka persistence.

It mainly uses Apache Ignite with Akka persistence to provide Journal and Snapshot stores by using the partitioned caches to benefit from distributed, highly available data grid features. Additionally, the nice query and data computation features in Ignite can be used to have normalized views from the event store and do analytical jobs over them. Note that it is advised to keep write nodes separate from read nodes for better scalability.

Final

We are using Akka version 2.5.7+ and Ignite version 2.3.0+.

Journal Plugin

  • All operations required by the Akka persistence Journal Plugin API are fully supported.
  • It uses an Apache Ignite partitioned cache with the default number of backups to 1 that can be changed into a reference.conf file.

Snapshot Store Plugin

How to Use

Enable the plugins in your Akka cluster configuration:

akka.persistence.journal.plugin = "akka.persistence.journal.ignite"
akka.persistence.snapshot-store.plugin = "akka.persistence.snapshot.ignite"

Configure Ignite data grid properties (default configured onlocalhost):

ignite {
  //to start client or server node to connect to Ignite data cluster 
  isClientNode = false
  // for ONLY testing we use localhost
  // used for grid cluster connectivity
  tcpDiscoveryAddresses = "localhost"
  metricsLogFrequency = 0
  // thread pools used by Ignite , should based into target machine specs
  queryThreadPoolSize = 4
  dataStreamerThreadPoolSize = 1
  managementThreadPoolSize = 2
  publicThreadPoolSize = 4
  systemThreadPoolSize = 2
  rebalanceThreadPoolSize = 1
  asyncCallbackPoolSize = 4
  peerClassLoadingEnabled = false
  // to enable or disable durable memory persistance
  enableFilePersistence = true
  // used for grid cluster connectivity, change it to suit your configuration 
  igniteConnectorPort = 11211
  // used for grid cluster connectivity , change it to suit your configuration 
  igniteServerPortRange = "47500..47509"
  //durable memory persistance storage file system path , change it to suit your configuration 
  ignitePersistenceFilePath = "./data"
}

After performing these steps, you will have Ignite enabled as your Journal and Snapshot Plugins. And you can enable it by starting the server node or client based on the configuration above.

Technical Details

For more details, feel free to dive into the code base — it is a small code base for now!

Summary 

This is a work in progress any contribution would be really helpful. Please check the project on GitHub and give a hand if you have any suggestions!

Hortonworks Community Connection (HCC) is an online collaboration destination for developers, DevOps, customers and partners to get answers to questions, collaborate on technical articles and share code examples from GitHub.  Join the discussion.

Topics:
akka ,apache ignite ,event sourcing ,big data ,data grid ,data persistence ,tck ,journal

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}