Over a million developers have joined DZone.

Developing a Modern Distributed System – Part I: Bootstrapping the Project

· Java Zone

Discover how powerful static code analysis and ergonomic design make development not only productive but also an enjoyable experience, brought to you in partnership with JetBrains

A few months ago, our performance and continuous delivery guild decided to gain more hands-on experience with distributed software architectures. As companies like Twitter or Netflix have open-sourced a lot of components from their software stacks this seemed a great idea to get started. A good introduction is a blog post about the Twitter software stack. However, we did not want to stare at architecture diagrams but rather getting our hands dirty and build something ourselves. Do you remember the good old days when Java Pet Store was new and fancy? We needed something similar for modern distributed architectures and finally settled to build a clone of the popular programmer Q&A site Stackoverflow: Hash-Collision was born. With Hash-Collision we want to address different issues in distributed systems such as:

  • Decomposition of the application into individual services
  • Bootstrapping of the whole system
  • Routing
  • Distributed service communication
  • Security among services
  • UI integration

As our lab idea sparked quite some interest at comSysto we now have multiple teams working on different aspects. In the first lab we concentrated on getting some minimalistic services running. We also decided on the JVM as a common runtime environment for all services to leverage the same tool chain and to simplify setup. However, at a later point we may decide to replace an existing service by a language that does not run on top of the JVM just for the sake of it.

System Structure

We decomposed Hash-Collision into fine-grained services which can be scaled independently. The system is decomposed into these verticals:

  • qa: handles discussions composed of questions and answers.
  • user: handles user data management
  • auth: handles credentials, authentication and authorization.

On top of these services is a UI which integrates services via their (REST) API endpoints: Hash-Collision's system structure


We want to provide completely self-contained services so WAR deployment wasn’t an option. After some research we stumbled across Spark and immediately liked its simplicity. Here is a echo server:

import static spark.Spark.*;

public class Echo {
    public static void main(String[] args) {
        get("/echo", (req, res) -> req.queryParams("msg"));

Each service provides some part of the domain logic, has its own data storage and exposes an API endpoint via HTTP. For simple deployment, it is contained in a single JAR that can easily be started from the command line.

Service Communication

For simplicity we settled for HTTP as communication protocol for API endpoints for the time being. However, in later versions we might switch to a binary protocol like Thrift. For asynchronous service communication we use AMQP with Msgpack encoding.


We decided to use Camel to implement a basic load balancing and routing component for the service layer. Service instances are registered dynamically by the router. A service instance announces itself by periodically sending heartbeat messages, so the router can keep track of running instances that are available. However, we currently run into availability problems when a service instance has just died and is not yet removed from the load balancer’s node list.



In Microservices: Decomposing Applications for Deployability and Scalability Chris Richardson describes different options for UI integration: Among them are calling services directly from the UI or using an API gateway. While calling services is easier to begin with, it can get unwieldy and a complex mess quite fast. Apart from that, the client issues many fine-grained HTTP requests which can decrease overall performance. Although Hash-Collision calls services directly from the UI due to implementation time constraints, we definitely recognize the benefits of an API gateway and will revisit the implementation later.

Service APIs and Shared Code

In every project there is a need to share code. The development team of the German retailer Otto describes in an article about their shop architecture (PDF) that they avoid company-internal shared code altogether. Instead they release such code always as open-source projects and integrate them. Due to the small amount of shared code, Hash-Collision currently uses a common project which is included in the build of individual service modules. In later incarnations we might as well separate the development cycles and may release shared code into an internal Maven repository. Shared code is also needed for service APIs. Service providers and consumers share a common API which also consists of transport objects. Some common approaches are:

  • Code duplication
  • Shared interface description language file and code generation
  • Shared code in the target language (client API)

Conclusion and Next Steps

In this lab we have built a very simple technical prototype. Especially operations-wise we have made quite some compromises such as simple local builds or provisioning via shell scripts. Next, another team will tackle:

  • Continuous delivery of services
  • Provisioning and system setup with tools like Vagrant or Docker

Development-wise we’d like to address:

  • Reduce the number of client requests by introducing an API gateway
  • Enable distributed tracing and monitoring to understand how requests flow through the overall system
  • Improve resilience
  • Add real service persistence

Stay tuned for more blog posts as we improve Hash-Collision.

Finally I would like to recommend a 2-days Training on Architecting for Continuous Delivery and Zero Downtime with Axel Fontaine, which will take place on 26th + 27th of January 2015 at comSysto. Read more about it by clicking at the link or sign up immediately HERE!

Learn more about Kotlin, a new programming language designed to solve problems that software developers face every day brought to you in partnership with JetBrains.


Published at DZone with permission of Comsysto Gmbh, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}