ActiveMQ in the Cloud

FuseSource just announced a public beta of new Enterprise products (Read more about it in Rob’s post). So it’s time to give you a bit more details on what we were working on for the past few months. One thing I want to emphasize in this post is how this project improves the experience of deploying and managing ActiveMQ brokers. Fuse Fabric (the central engine behind Fuse Enterprise) can help you provision and manage your brokers better and additionally help with cloud deployments.

Provisioning

While classic server software (ActiveMQ included) deployment scenarios, which include unpacking a distro, setting a config and starting/stopping software, works OK for small deployments, there are a lot of challenges people encounter when trying to set and manage a large number of instances. Here are some of them:

• Enormous amount of work needed to set everything – imagine a large cluster you need to set, ssh-ing, unpacking, copying and tweaking config files is very tedious and error-prone process
• Changing configuration during runtime – you need again to manually tweak every file and restart the server, which is all but fun
• Upgrading – can also be a challenge and time consuming

Some of the things folks usually do to make their life easier is to:

• Keep XML config part of the spec as a template and in version control system and tweak as much things as possible with properties – this makes it easier to keep things under control and minimize the potential of an error when managing a configuration on large number of instances.
• Keep configuration separate from distribution to make it all easier to upgrade
• Use tools like Puppet or Chef to even further make things easier in these scenarios

One of the things where Fabric excels is centralized configuration management. In Fabric it’s really easy to spin new container instances on your servers (using ssh) or to the public cloud. Additionally it’s really easy to deploy ActiveMQ broker to those containers,
by simply applying appropriate profile to them. So generally, you can predefine a template for your broker, like XML configuration template and all necessary properties in a profile
and then with a simple Karaf command (or even mouse click in our tools, like FuseIDE and Fuse Management Console) deploy as many instances as you like of that broker (no matter if it’s a physical box in your data center or a VM at the public could provider).

Discovery

Fabric uses Apache ZooKeeper as it’s central registry of container instances running. This same registry is used to keep track of all brokers running inside a Fabric instance. This means that we can use this registry to discover all brokers inside a certain group. Therefore we created a new discovery protocol (called fabric of course) that can do that. So clients can connect to the broker group (think of it as a cluster) without any need to know exact location of the brokers. So now you can see how Fabric helps deploying your brokers to the cloud. First, you can use its ability to start containers (and deploy brokers on them) on any server or cloud provider available. And than clients can use Fabric to discover brokers and connect to them, all location agnostic.

Topologies

ZooKeeper as a central registry allows us to do some more nice things in domain of broker topologies. For example, currently the master-slave topology of ActiveMQ brokers depends on shared storage (either shared file system or enterprise JDBC database). In this scenario the master election and slave locking depends on the locking ability of the shared storage, limiting it to certain type of hardware. With ZooKeeper as a distributed registry, battle-proven for this kind of use cases, it’s easy to create a master-slave topologies with master election done on ZooKeeper locks. If you want to have a persistent master-slave, you still gonna need to store your messages in a some kind of a shared storage, but locking is not store’s job any more. And on the other hand it’s really easy to create non-persistent cluster of brokers with shared-nothing philosophy. Creating a master-slave is as simple as creating a multiple brokers with the same name in the group. The first one started will become a master, while others will be slaves waiting for the master to fail.

Another example of enhanced topology possibilities with ActiveMQ and Fabric is new ways you can set networks of brokers. Just as clients can use ZooKeeper registry to discover brokers, network connectors can use the same discovery protocol to connect the broker with all other brokers in the certain group. Again, brokers are mutually totally location agnostic, which is something desired in deployment scenarios with modern infrastructure. Could it be done any easier? Finally, you might want to mix these two topologies and create connected networks of master/slave broker, which with Fabric, is just easy to do as those basic topologies.

Upgrades and Patching

And now you say you need to upgrade all of the dozen brokers you have? No problem, with Fabric you can centrally update any bundle you’re using in your profile and that change will be propagated to all brokers. Now it’s time to talk about risk management of upgrades and this is where profile versions come into play. Instead of messing with profiles that are used in production at the moment, you can create a new version of it with updated versions of all bundles you plan to use. Once you have that profile ready, you can test it out on one or a few instances. And only after you’re certain that everything is OK, you can apply a new profile to all brokers. Of course, it’s easy to rollback to the previous version if anything goes wrong.

Fabric also provides you with the mechanism to provide a simple incremental patch jar with only certain classes modified (a single bug-fix). These classes will be then used instead of old ones and will allow us to provide a fix for a critical bug without need to upgrade the project version. This can be useful for dealing with critical production bugs, where waiting for the next release is not an option.

More resources

This post was just meant to give you a glance view of what’s coming from FuseSource regarding easier deployment, provisioning and managing of your messaging (and generally integration) infrastructure. You can check some more docs on the projects. Especially, there’s some more info that explains ActiveMQ concepts explained here in more details and shows how to run examples. You will definitely hear more from us on this topic in coming days, so stay tuned.