Over a million developers have joined DZone.

Replication as a Service (Part One)

DZone 's Guide to

Replication as a Service (Part One)

· Cloud Zone ·
Free Resource

[This article originally written by DeWayne Filppi.]


Cloud computing vendors provide a variety of APIs for deploying and interacting with the services they provide.  These APIs have grown in number along with the variety of services being offered.  Initially, the ability to request and manipulate cloud compute resources was provided.  This has grown to include many other infrastructure level services such as load balancing, storage, identity, DNS, images, and networking.  From there APIs have moved more into platform level services including blob storage, backup, queuing, caching, databases, big data and many more.  In a sense, Cloudify unifies several of these services behind a declarative (recipe) facade.   This allows users to interact with the myriad APIs by interacting with Cloudify’s generic API (in the case of block storage) or recipes in a portable way.  The goal of creating a replication as a service capability goes in a different direction from standardizing existing APIs, to creating a standardized cloud-neutral (and potentially inter-cloud) service that is not provided by the cloud vendors themselves. Replication as a service is built upon recent work that brought Gigaspaces XAP to the cloud.  Using the existing xap-gateway recipe, the construction of a replication network is possible by hand.  This post, and ones to follow, describe the process of turning this basic capability into a true cloud service, complete with multi-cloud support and REST API.

The Initial Project Goal

I find it useful to start such projects by focusing on the end user experience (in the case the API user), and worry about details later.  For replication as a service (RAAS), I considered an arbitrarily complex replication network topology.  XAP WAN replication capability allows for arbitrarily complex topologies, so it is the ideal tool for a general purpose replication service.  To focus the effort, I decided to define and implement a single (but complex) use case: define and deploy a multi-cloud replication network using a REST API. When done manually in a non-cloud environment, setting up a XAP replication network is a well defined process.  Gateway processing units are hosted on nodes at each site that have internet access, and configured with the other site’s addresses.  In some cases, NAT mapping comes into play, which XAP handles nicely (although not nicely enough for the cloud as we’ll see soon).  The different site’s information can be assembled in advance, and all configurations done in an orderly fashion.  Attempting the same on the cloud requires far more dynamism.

To present replication as a service, one precondition exists for any Cloudify oriented approach: Cloudify itself must be running on each site that will participate.  I placed a further simplifying precondition, and that is that each site must run the same version of Cloudify.  The deployment/boostrapping of Cloudify itself must be done manually, although I suppose one could imagine “Cloudify as a service”, but that seems one abstraction too far, and beyond the scope of this effort.  One other clarification is needed as well; the creation of a Cloud based replication network as a service requires deploying XAP itself along with the space that holds the data. One approach (and therefore user experience) I considered was emulating the manual method at the API level.  With this approach, the user would call the REST API on each site with the relevant configuration supplied in the gateway recipe.  One downside of this approach is it ties gateway pu’s to gateway hosts (recall that every Cloudify service recipe represents a Cloud machine).  The other big problem is it can’t be done (at least without DNS), because when I try to deploy site 1 first, I don’t know anything about other sites in my topology because they don’t exist yet.  This led me to a two-phase approach, where the gateway node is deployed first (via conventional Cloudify recipe), and a custom command is used to actually create the topology.  We’ll revisit the single phase approach later, as it becomes technically possible as a side effect of solving different problems.  Whether it is desirable is another issue. To recap, at this point the approach was going to be:

  1. Deploy Cloudify at every site.
  2. Call the REST API to deploy XAP including a gateway node on every site.
  3. Call the REST API to invoke the custom command to deploy the replication spaces and gateway PUs at every site.
What the user winds up with is one REST call for each site to deploy the Cloudify recipes, and then another call for each site to deploy the actual space and gateway.  Not only this, but information from the first call would need to be retained (by the caller) from the first pass to send on the second pass to configure the components.  I could have proceeded with this approach as a “version 1.0″ , but it didn’t feel right.  It is too low level and too complex, and would have required DNS.  A user of the API shouldn’t need to understand XAP, gateways, or anything else.  A user needs to replicate data from point A to point B (and maybe lots of other points); the rest (no pun intended) is details.

So going back to square one and imagining planning the replication, what would I do?  I’d draw circles on a napkin with arrows between
them representing sites.  In order to get as close to this (without writing a GUI) as possible, I changed the approach to one of:
  1. Deploy Cloudify at every site. (can’t get away from this)
  2. Define my topology (a step not included in the previous list but obviously a prerequisite).
  3. Render the topology in a single REST call across all sites.

The idea is I take my napkin and hand it to the API to worry about the details.   In the parlance of REST, the napkin is JSON.  In order to have a single REST call do everything, the REST API must in effect act as a meta-Cloudify, deploying recipes and calling custom commands on all sites from a single location.  This feels cleaner because it means the topology itself has a model (abstraction) that can be rendered and updated as time goes by, without having to dig into implementation details.

The Topology Definition

They key abstraction in the RAAS implementation is the replication topology, which will be expressed in JSON to be compatible with REST norms.  The topology definition must contain sufficient information to complete the topology construction on all sites.  The first order of business is defining the sites themselves.  Each site needs a symbolic name and a URL that designates the Cloudify endpoint at the site (recall that multiple Cloudify installations will be enlisted).  In JSON, this looks something like:

  {"site1-name":"<some public ip>:8100", "site2-name":"<some public ip>:8100" ... } 

In order to support multiple simultaneous topologies, a topology needs a name to refer it.  This expands the definition a bit with a “name” entry:

    {"site1-name":"<some public ip>:8100", "site2-name":"<some public ip>:8100" ...  }

To represent the directional flow of data,  I chose the concept of an edge from graph theory to represent a unidirectional connection between two sites.  By listing all edges, an arbitrarily complex topology can be built up as a list of 2 element tuples.   For example, a simple master-slave edge between sites ‘site1′ and ‘site2′  would be represented in JSON as


.  A bidirectional connection would look like


.  This makes the example look like:

    {"site1-name":"<some public ip>:8100", "site2-name":"<some public ip>:8100" ...  }

 That covers the basics.  One (of many) limititations of this initial attempt is the ignoring of cloud security issues (i.e. security groups).  Currently, it is assumed that such issues are handled externally (either manually, or by overrides to the underlying Cloudify recipes).  In any case, one feature of XAP containers that host replication gateways, is that they have to be started with the ports used for replication.  Since the REST API starts these containers, it must have port numbers defined in the topology definition for each site.  These could have some kind of default values, but the ports will have to be open both in the container and to the internet.  To accommodate this, there is another section in the JSON called “ports”, that defines the ports for each site.  Adding this, the JSON almost complete:

    {"site1-name":"<some reachable ip>:8100", "site2-name":"<some reachable ip>:8100" ...  }

The pretty much covers the topology definition.  There are some other settings associated with tuning the actually replication performance (i.e. batch size, send interval, etc…) but I won’t delve into these now to avoid muddying the water.

The REST Endpoint

Since the objective is constrained to just the “create replication network” use case, there is only a single verb in the REST API so far.  The REST API is implemented with Spring MVC.  The API defines a single method that accepts  a POST action with the name of the topology network as the path variable, example:


It’s a single action that does quite a lot, so it’s worth going through the logic at a high level.  All of the topology information is included in the POST body as describe above.  For a REST call that aspires to do so much (and potentially take a long time to do it), request validation is critical.  More generally, failing fast (or as fast as possible) is critical.  A limited amount of this has been implemented, but it is definitely an area that could be expanded.  Once the request is validated, each participating site is contact concurrently when possible to minimize the call latency.  I use the term “minimize” loosely, because any single call that is spinning up multiple VMs on multiple Clouds (potentially), is not going to be quick.  The execution of the REST call causes the following actions:

  1. Request validation.  The syntax of the request is checked.  The presence of required fields is checked as well as the validity of scalar entries.
  2. Participant validation.  Each participating Cloudify site is contacted to verify availability.  Each site is check to make sure a topology with the same name is not running already.
  3. XAP is deployed at each site.  This includes a manager, container(s), and gateway.  The gateway container is started with correct ports open per the topology definition.  The gateway is deployed with a custom NAT mapper, which permits dynamic configuration (unlike the standard version).  Data containers and gateway containers are deployed in separate zones so that future gateway and space deployments can be properly targeted.  This operation is done in parallel across all sites.
  4. Spaces to hold replication data are created and deployed at every site.  The space is given the same name as the topology.  Space specifics such as partitions and backups are taken from the topology definition.  The space is deployed with the local gateway name taken from the topology, and an empty targets block for later dynamic target introduction. This operation is done in parallel across all sites.
  5. Replication gateways are deployed.  A gateway processing unit is constructed at each site and deployed, and the space target is updated.   Delegators and sinks are derived from the topology definition (edges).  Lookups are constructed from previous steps in the deployment process, where the needed IP addresses are assigned.  The dynamic NAT mapping table is updated from private/public IP information gathered from the previous gateway deployment step. This operation is done in parallel across all sites.

When all of this completes (partial failures cause a complete rollback), the system begins replicating.


Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}