Running Axon Server in Docker
Running Axon Server in Docker
Axon server in Docker, both using the public image up on Docker Hub as well as with a locally built image, and why you might want to do that.
Join the DZone community and get the full member experience.Join For Free
In my previous post, I showed you how to run Axon Server locally and configure it for secure operations. We also looked at the possibilities for configuring storage locations. This time around we’ll look at running it in Docker, both using the public image up on Docker Hub as well as with a locally built image, and why you might want to do that.
Note: We now have a repository live on GitHub with scripts, configuration files, and deployment descriptors. You can find it on https://github.com/AxonIQ/running-axon-server.
Axon Server in a Container
Running Axon Server in a container is actually pretty simple using the provided image, with a few predictable gotchas. Let’s start with a simple test:
Ok, anyone who ever ran Docker containers before would see that coming: the container may have announced that ports 8024 and 8124 are to be exposed, but that was just a statement of intent. So we ^C ourselves out of here and add “
-p 8024:8024 -p 8124:8124”. On the Axon Server side nothing looks different, but now we can get access:
As discussed last time, having a node name “
87c201162360” is no problem, but the hostname will be, as a client application will by default follow Axon Server’s request to switch to that name without question, and fail to find it. We can reconfigure Axon Server without much trouble, but let me start with telling a bit about the container’s structure.
It was made using Axon Server SE, which is Open Source and can be found at https://github.com/AxonIQ/axon-server-se. The container is built using a compact image from Google’s “distroless” base images at the gcr.io repository, in this case “
gcr.io/distroless/java:11”. The application itself is installed in the root with a minimal properties file:
/data” and “
/eventdata” directories are created as volumes, and their data will be accessible on your local filesystem somewhere in Docker’s temporary storage tree. Alternatively, you can tell docker to use a specific directory, which will allow you to put it at a more convenient location. A third directory, not marked as a volume in the image, is important for our case: If you put an “
axonserver.properties” file in “
/config”, it can override the settings above and add new ones:
Now if you query the API (either using the “
docker logs” command to verify startup has finished, or simply repeating the “
curl” command until it responds) it will show that it is running with name “
axonserver” and hostname “
localhost”. Also, if you look at the data directory, you will see the ControlDB file, PID file, and a copy of the log output, while the “
events” directory will have the event and snapshot data.
From Docker to Docker-Compose
Running Axon Server in a Docker container has several advantages, the most important of which being the compact distribution format: With one single command we have installed and started Axon Server, and it will always work in the same, predictable, fashion. You will most likely use this for local development and demonstration scenarios, as well as for tests of Axon Framework client applications.
That said, Axon Server is mainly targeted at a distributed usage scenario, where you have several application components exchanging Events, Commands, and Queries. For this you will more likely employ docker-compose, or larger-scale infrastructural products such as Kubernetes, Cloud-Foundry, and Red Hat OpenShift.
To start with docker-compose, the following allows you to start Axon Server with “
./events”, and “
./config” mounted as volumes, where the config directory is actually Read-Only. Note: This has been tested on MacOS and Linux. On Windows 10 named volume mapping using the “
local” driver will not work, so you need to remove the “
driver” and “
driver_opts” sections in the file below.
The new Windows Subsystem for Linux (WSL version 2) in combination with the Docker Desktop based on it will hopefully bring relief, but for the moment you will not be able to use volumes in docker-compose and then access the files from the host on Windows.
This also sets the container’s hostname to “
axonserver”, so all you need to add is an “
Now you have it running locally, with a fresh and predictable environment, and easy access to the properties file. Also, as long as you leave the “
data” and “
events” directories untouched, you will get the same event store over subsequent runs, while cleaning them is simply a matter of removing and recreating those directories.
Differences With Axon Server EE
To extend this docker-compose setup to Axon Server EE, we first need to build an image, as there is no public image for it. Using the same approach as with SE however, it is a relatively simple one that will work for OpenShift, Kubernetes, as well as Docker and docker-compose. Also, we can be a bit more security conscious and run Axon Server as a non-root user. This last bit forces the usage of a two-stage Dockerfile, since the Google “
distroless” images do not contain a shell, and we want to run a few commands:
The first stage creates a user and group named “
axonserver”, as well as the directories that will become our volumes, and finally sets the ownership. The second stage begins by copying the account (in the form of the “
passwd” and “
group” files) and the home directory with its volume mount points, carefully keeping ownership set to the new user. The last steps are the “regular” steps, copying the executable jar and a common set of properties, marking the volume mounting points and exposed ports, and specifying the command to start Axon Server.
For the common properties we’ll use just enough to make it use our volume mounts, and add a log file for good measure:
You can build this image and push it to your local repository, or keep it local if you only want to run it on your development machine. On the docker-compose side we now can specify three instances of the same container image, using separate volumes for “
events”, and “
log”, but we haven’t provided it yet with a license file and token. We’ll use secrets for that:
All three files will be placed in the “
config” directory using a “
secrets” section in the service definition, with an environment variable added to tell Axon Server about the location of the license file. As an example, here is the resulting definition of the first node’s service:
Note that for “axonserver-2” and “axonserver-3” you’ll have to adjust the port definitions, for example using “8025:8024” and “8026:8024” for the first port, to prevent all three trying to claim the same host-port. The properties file referred to in the secrets’ definition section is:
Just like last time we enable auto-clustering and access control, with a generated token for the communication between nodes. The “
axonserver-token” secret is used to allow the CLI to talk with nodes. A similar approach can be used to configure more secrets for the certificates, and so enable SSL.
Kubernetes and StatefulSets
Kubernetes has quickly become the “de facto” solution for running containerized applications on distributed infrastructure. Thanks to its API-first approach it allows for flexible deployments using modern “infrastructure as code” design patterns. Due to the tight integration possible with Continuous Integration pipelines, it is also perfect for “ephemeral infrastructure” testing, with complete environments set up and torn down with minimal work. It is also the go-to platform for microservices architectures, and I think I have collected enough bonus points in the buzzword bingo with it.
All jokes aside, many of our customers deploy their applications on Kubernetes clusters, and we get regular questions about “the best way” to run Axon Server on it. With a platform like Kubernetes, you’ll find that there are a lot of customization points, but they are all subject to the underlying deployment model, which is (preferably) that of a horizontally scalable and stateless (micro-)service, where the lifecycle is easily automatable.
For Axon Server Standard Edition, scalability is vertical, as it has no concept of a clustered deployment. Even stronger, a running Axon Server instance is definitely stateful due to the event store. So for now let’s focus on the most important aspect of a Kubernetes deployment of Axon Server: fixing the server’s identity and persistence using StatefulSets.
As stated before, an Axon Server instance has a clear and persistent identity, in that it saves identifying information about itself and (in the case of Axon Server EE) other nodes in the cluster, in the controlDB. Also, if it is used as an event store, the context’s events will be stored on disk as well, and whereas a client application can survive restarts and version upgrades by rereading the events, Axon Server is the one providing those.
In the context of Kubernetes that means we want to bind every Axon Server deployment to its own storage volumes, and also to a predictable network identity. Kubernetes provides us with a StatefulSet deployment class which does just that, with the guarantee that it will preserve the automatically allocated volume claims even if migrated to another (k8s) node.
The welcome package downloaded in part one includes an example YAML descriptor for Axon Server, which I have included below (with just minor differences):
Highlighted in the listing above are two important lines for Axon Server SE, the first telling Kubernetes we want only a single instance, the second referring to the SE container image. Important to note here is that this is a pretty basic descriptor in the sense that it does not have any settings for the amount of memory and/or CPU to reserve for Axon Server, which you may want to do for long-running deployments, and it “just” claims 5GiB of disk space for the Event Store. Also, we have not yet provided any means of adjusting the configuration. To complete this we need to add Service definitions that expose the two ports:
Now you’ll notice the HTTP port is exposed using a LoadBalancer, while the Service for the gRPC port has a defaulted type of “
ClusterIP” with “
clusterIP” set to “
none”, making it (in Kubernetes terminology) a Headless Service. This is important because a StatefulSet needs at least one Headless Service to enable DNS exposure within the Kubernetes namespace.
Additionally, client applications will use long-living connections to the gRPC port, and are expected to be able to explicitly connect to a specific node. Apart from that, the deployment model for the client applications is probably what brought you to Kubernetes in the first place, making an externally accessible interface less of a requirement. The client applications will be deployed in their own namespace and can connect to Axon Server using k8s internal DNS.
The elements in the DNS name are (from left to right):
- The name of the StatefulSet, a dash, and a sequence number (starting at 0). You’ll recognize this as the Pod’s name.
- The name of the service.
- The name of the namespace. (“
default” if unspecified)
If you want to deploy Axon Server in Kubernetes, but run client applications outside of it, you actually can use a “LoadBalancer” type service since gRPC uses HTTP/2, but you will need to fix it to the specific pod using the “
statefulset.kubernetes.io/pod-name” selector and the Pod’s name as value, and repeat for all nodes. However, as this is not recommended practice we’ll not go into that.
Differences With Axon Server EE
There are several ways we can deploy a cluster of Axon Server EE nodes to Kubernetes. The simplest approach, and most often correct one, is to use a scaling factor other than 1, letting Kubernetes take care of deploying several instances. This means we will get several nodes that Kubernetes can dynamically manage and migrate as needed, while at the same time fixing the name and storage. As we saw with SE you’ll get a number suffixed to the name starting at 0, so a scaling factor of 3 gives us “axonserver-0” through “axonserver-2”.
Of course we still need a secret to add the license file, but when we try to add this to our mix we run into a big difference with docker-compose: Kubernetes mounts Secrets and ConfigMaps as directories rather than files, so we need to split license and configuration to two separate locations. For the license secret we can use a new location “
/axonserver/license/axoniq.license” and adjust the environment variable to match. For the system token we’ll use “
/axonserver/security/token.txt”, and for the properties file we’ll use a ConfigMap that we mount on top of the “
/axonserver/config” directory. We can create them directly from their respective files:
In the descriptor we now have to announce the secret, add a volume for it, and mount the secret on the volume:
Then a list of volumes has to be added to link the actual license and properties:
It is arguable that the properties should also be in a secret, which tightens up security on the settings in there, but I’ll leave that “as an exercise for the reader.”
Now there is only one thing left, and that has to do with the image we built for docker-compose. If we try to start the StatefulSet with 1 replica just to test if everything works, you’ll find that it fails with a so-called “CrashLoopBackoff”. If you look at the logs, you’ll find that Axon Server was unable to create the controlDB, and that is odd given that it worked fine for SE.
The cause is a major difference between plain Docker and Kubernetes, in that volumes are mounted as owned by the mount location’s owner in Docker, while Kubernetes uses a special security context, defaulting to root. Since our EE image runs Axon Server under its own user, it has no rights on the mounted volume other than “read”.
The context can be specified, but only through the user or group’s ID, and not using their name as we did in the image, because that name does not exist in the k8s management context. So we have to adjust the first stage to specify a specific numeric value, and then use that value in the security context:
Now we have an explicit ID (1001 twice) and can add that to the StatefulSet:
With this change we can finally run Axon Server successfully, and scale it up to the number of nodes we want. However, when the second node comes up and tries to register itself with the first, another typical Kubernetes behaviour turns up, when we see the logs of node “
axonserver-enterprise-0” filling up with DNS lookup errors for “
axonserver-enterprise-1”. This is caused by the way the StatefulSet Pods get added to the DNS registry, which is not done until the readiness probe is happy. Axon Server itself is by then already busily running the auto-cluster actions, and node 0 is known even though the way back to node 1 is still unknown.
In a Pod migration scenario, if e.g. a k8s node has to be brought down for maintenance, this is exactly the behaviour we want, even though confusing when we see it here during cluster initialisation and registration. If you want, you can avoid this by simply not using the auto-cluster options, and doing everything by hand, but given that it really is a “cosmetic” issue and in no way has lasting effects, you can also simply ignore the errors.
Alternative Deployment Models
Using the scaling factor on our StatefulSet is pretty straightforward, but does have a potential disadvantage: we cannot disable (shutdown if you like) a node without also disabling all higher numbered nodes. If we decide to give the nodes different roles in the cluster, define contexts that are not available on all nodes, or want to bring a “middle” node down for maintenance, you run into the horizontal scaling model imposed by Kubernetes.
It is possible to do individual restarts, simply by killing the Pod involved, which will prompt Kubernetes to start a new one for it, but we cannot shut it down “until further notice”. The assumption made by the StatefulSet is that each node needs its storage and identity, but they all provide the same service. If you want to reduce the scaling by one, the highest numbered one will be taken down.
=Taking the whole cluster down for maintenance is easy, but that is not what we want. An alternative model is to create StatefulSets per role, with as ultimate version a collection of single-node sets. This may feel wrong from a Kubernetes perspective, but works perfectly for Axon Server.
In the first installments we discussed the different storage settings we can pass to Axon Server. In the context of a Docker or Kubernetes deployment this poses a double issue: As a kind of obvious first one, we want to ensure that the volume is persistent, and that we have direct access to it to enable us to make backups. However there is an additional issue that has to do with the implementation of the volume, in that it needs to be configurable so we can extend it when needed.
For Docker and docker-compose it is quite possible to do this on Windows, but not with the easiest implementation of the “local” driver. Kubernetes on a laptop or desktop however is a very different scenario, where practically all implementations use a local VM to implement the k8s node, and that VM cannot easily mount host directories. So while this will work for an easily created test installation, if you want a long-running setup under Windows I would urge you to look at running it as a local installation.
In the cloud, both AWS and Google allow you to use volumes that can be extended as needed, without the need for further manual adjustments. As we’ll see in the next installment, for VMs you will be using disks that may need formatting before they can be properly mounted. A newly created volume in k8s is immediately ready for use, and resizing is only a matter of using the Console or CLI to pass the change, after which the extra space will be immediately usable.
In the next installment we’ll be moving to VMs, and touch on a bit more OS specifics as we build a solution that can be used in a CI/CD pipeline, with a setup that does not require manual changes updates and version upgrades.
The example scripts and configuration files used in this blog series are available from GitHub! Please visit https://github.com/AxonIQ/running-axon-server to get a copy.
Opinions expressed by DZone contributors are their own.