Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Service Discovery: More Than it Seems (Part 1)

DZone's Guide to

Service Discovery: More Than it Seems (Part 1)

Let's dive into the ins and outs of service discovery, specifically client-side service discovery, using Mesosphere Marathon and Spring Cloud.

· Cloud Zone
Free Resource

Are you joining the containers revolution? Start leveraging container management using Platform9's ultimate guide to Kubernetes deployment.

Upon transition to distributed systems with a large number of instances of services, there are problems with their discovery and load balancing between them. As a rule, solving these problems means using specific solutions like Consul, Eureka, or good old Zookeeper, with Nginx, HAProxy, and some bridge between them (see registrator).

The main problem with this approach is that this is a huge number of different integrations, and, as a consequence, that means more failure points where something might go wrong. Because in addition to the above solutions, surely local, small (or not small) PaaS (for example Mesosphere Marathon or Kubernetes) will be used. The latter, by the way, already stores the necessary information about the environment (because all deployment goes through them). And we should ask a question: "Could we not use these separated solutions for service discovery and instead reuse Marathon or another orchestrator for solving this problem?"

The short answer "Yes, we can."

Disposition

Ok, try to look what we have:

  • Apache Mesos and its faithful framework Marathon are used for service orchestration (deployment, scaling, etc.)
  • Some services are written with Spring Boot and its extension Spring Cloud

Mesos without sugar (read as without frameworks) is a cluster resource management system that can be extended by frameworks. Frameworks solve different problems. Some of them can launch short-lived tasks (Chronos) for batching and processing data, for example. Others launch long-lived tasks (Marathon) or services that are processed requests. And to add even more, appropriate frameworks for Hadoop or Jenkins exist.

Mesosphere Marathon is the very same framework. It can launch, stop, restart, scale, and do other things that are required to manage long-lived tasks or services.

Spring Cloud is a framework, but it's for developing these services. It has an implementation of basic patterns for distributed systems and specific integrations with different service registries or configuration management systems like Consul or others.

In Spring Cloud, there are two different implementations for service discovery problem-solving.

First, Netflix Zuul implements a server-side service discovery pattern. The main idea is that several smart routers have information about services and their locations and different meta information about instances. These routers provide a limited number of static HTTP-resources that are used as proxies by clients. If we don't consider Spring, then we may say Nginx is a classic router because of its dynamic configuration capabilities. The pattern is presented as an image below.

Image title


The second implementation is called client-side service discovery. Its main difference from server-side service discovery is that there is no router or any additional failure point. Instead of a smart router, a smart client load balancer is used. It is smart because it has information about balanced services that are needed to be called and it uses statistics for decision-making. Spring Cloud has the load balancer called Netflix Ribbon. And the pattern is presented as an image below.

Image title


In this series of articles, we will talk about the client-side variant mostly, but we will touch on server-side too.

@EnableDiscoveryClient

In Spring, all the things (or almost all) start working with one or several dozen annotations over classes or methods or variables. Also, some configuration in YAML files might be required, or some environment parameters.

From the reference, we know that the magic annotation @EnableDiscoveryClient over the main class of an application enables (after configuration modification and starter inclusion) service discovery features. At least locally inside of our application. Easy peasy:

@SpringBootApplication
@EnableDiscoveryClient
public class Application {
    public static void main(String[] args) {
        SpringApplication.run(Application.class, args);
    }
}


After that, the magic without sorcery is happening. Spring finds this annotation and loads all configurations that are described in META-INF/spring.factories as loadable for EnableDiscoveryClient:

org.springframework.cloud.client.discovery.EnableDiscoveryClient=\
org.springframework.cloud.xxx.discovery.XXXDiscoveryClientConfiguration


What configurations could be loaded? And where can their location be found? For an answer to this question, we need to consider the fact that Spring Cloud consists of the base part and connectors from Spring Boot starters. The base part has an implementation of common patterns, has common beans, and a common configuration. Connectors, just the opposite, have a particular implementation of concrete third-party solutions.

Image title

Let's say Netflix Eureka is added to the dependencies, then in the classpath, there will be one configuration factory. If there is a starter for Consul, there will be another.

But there is one sad thing. The annŠ¾tation by itself, in the case of production (not helloworld-development), is unusable because the right properties should be written in its proper place: bootstrap.yml. And each connector has its own "right" properties.

All of the service registries, and Eureka, and Consul, and Marathon, have different features and different internals. There are at least different kinds of connection to them, APIs, and specific features like DNS-discovery. Universal configuration is not possible, or at least very hard to achieve. And moreover, it is not necessary.

Let's go one step back to the configuration that is enabled by @EnableDiscoveryClient. And first, you might Google, or find out through a search in your favorite IDE, the implementation of the DiscoverClient interface. The primary (on the face of it, by the way) interface looks like:

public interface DiscoveryClient {
    public String description();
    public ServiceInstance getLocalServiceInstance();
    public List<ServiceInstance> getInstances(String serviceId);
    public List<String> getServices();
}


Everything is pretty obvious. We could get a description for HealthIndicator. We could get ourselves. We could fetch instances of services with a particular identifier. And, at last, we could receive all registered services (more precisely, its identifiers).

It's time to implement an interface for fetching data from Marathon.

First Blood

But how to fetch the data? It is the first problem that we need to solve. And it is not hard.

First, it has a powerful API. And second, there is a Java SDK.

Let's fetch the service ids:

@Override
public List<String> getServices() {
    try {
        return client.getApps()
            .getApps() //more apps for god of apps
            .parallelStream()
            .map(App::getId) //fetch identifiers
            .map(ServiceIdConverter::convertToServiceId) //some magic ;)
            .collect(Collectors.toList());
    } catch (MarathonException e) {
        return Collections.emptyList();
    }
}


No magic except ServiceIdConverter::convertToServiceId. What a strange converter! We need to dive deeper to the internal representation of service identifiers in Marathon. In general, they have the following pattern: /group/path/app but a symbol, /, cannot be used as part of a virtual host because of the HTTP specification. And some parts of Spring Cloud, where a service identifier is used as a virtual host, will not work. So instead of /, we will use a separator that is allowed to be in a hostname. Yes, you are right. It is a point. And we need a mapping between these two representations: /group/path/app and group.path.app. And the magic converter does this job.

Fetching instances by service id is also not so hard:

@Override
public List < ServiceInstance > getInstances(String serviceId) {
    try {
        return client.getAppTasks(ServiceIdConverter.convertToMarathonId(serviceId))
            .getTasks()
            .parallelStream()
            .filter(task - > null == task.getHealthCheckResults() ||
                task.getHealthCheckResults().stream().allMatch(HealthCheckResult::isAlive)
            ).map(task - > new DefaultServiceInstance(
                ServiceIdConverter.convertToServiceId(task.getAppId()),
                task.getHost(),
                task.getPorts().stream().findFirst().orElse(0),
                false
            )).collect(Collectors.toList());
    } catch (MarathonException e) {
        log.error(e.getMessage(), e);
        return Collections.emptyList();
    }
}


The main thing we need to check is that all services' health checks are passing: HealthCheckResult::isAlive, because we want to work only with healthy instances. The health checking feature is provided by Marathon itself. It has settings for setting up health checks, and it checks them with some interval. All that information could be fetched from the API.

We should choose only one port (basically the first):

task.getPorts().stream().findFirst().orElse(0).

Wait, wait, wait, you might have said. What if the service has several ports? Unfortunately, we have a limited number of variants. On the one hand, we should return an object that implements the ServiceInstance interface that has the getPort method, which returns only one port, as you may guess. On the other hand, we don't know which ports are used. Marathon doesn't give any information about it, so we simply take port that is defined first. Maybe luck.

This problem may be solved in a registrator-like way. The solution is to use any port in a service identifier like that: group.path.app.8080 in case of multiple ports.

We are a little distracted. It's time to define our implementation as a bean:

@Configuration
@ConditionalOnMarathonEnabled
@ConditionalOnProperty(value = "spring.cloud.marathon.discovery.enabled", matchIfMissing = true)
@EnableConfigurationProperties
public class MarathonDiscoveryClientAutoConfiguration {
    @Autowired
    private Marathon marathonClient;

    @Bean
    public MarathonDiscoveryProperties marathonDiscoveryProperties() {
        return new MarathonDiscoveryProperties();
    }

    @Bean
    @ConditionalOnMissingBean
    public MarathonDiscoveryClient marathonDiscoveryClient(MarathonDiscoveryProperties discoveryProperties) {
        MarathonDiscoveryClient discoveryClient =
            new MarathonDiscoveryClient(marathonClient, marathonDiscoveryProperties());
        return discoveryClient;
    }
}


Let's go over what's important here. First, we use conditional annotations: @ConditionalOnMarathonEnabled  and @ConditionalOnProperty. So if the feature is turned off through the spring.cloud.marathon.discovery.enabled property, then configuration would not be loaded. Second, the annotation @ConditionalOnMissingBean is placed under the client bean that gives the end user an opportunity to replace the client bean in its app.

We need to do just a little more. Let's configure a Marathon client. Naive, but working, the implementation looks that:

spring:
cloud:
marathon:
scheme: http #url scheme
host: marathon #marathon host
port: 8080 #marathon port


For reading these properties, we need a configuration properties bean:

@ConfigurationProperties("spring.cloud.marathon")
@Data //lombok is here
public class MarathonProperties {
    @NotNull
    private String scheme = "http";

    @NotNull
    private String host = "localhost";

    @NotNull
    private int port = 8080;

    private String endpoint = null;

    public String getEndpoint() {
        if (null != endpoint) {
            return endpoint;
        }
        return this.getScheme() + "://" + this.getHost() + ":" + this.getPort();
    }
}


And our configuration is very similar to the previous configuration:

@Configuration
@EnableConfigurationProperties
@ConditionalOnMarathonEnabled
public class MarathonAutoConfiguration {
    @Bean
    @ConditionalOnMissingBean
    public MarathonProperties marathonProperties() {
        return new MarathonProperties();
    }

    @Bean
    @ConditionalOnMissingBean
    public Marathon marathonClient(MarathonProperties properties) {
        return MarathonClient.getInstance(properties.getEndpoint());
    }
}


After that, we might go to our app and autowire the DiscoveryClient  bean:

@Autowired
private DiscoveryClient discoveryClient;


And, for example, fetch a list of instances for individual services:

@RequestMapping("/instances")
public List<ServiceInstance> instances() {
    return discoveryClient.getInstances("someservice");
}


But at this moment we have the first surprise. In the real world, our goal is not to fetch instances because we want to fetch them. We want to call them and load balance our calls between instances. Sad but true, DiscoveryClient is not used for load balancing, at least implicitly. Ok, I know that it is used for dynamic registration in edge server implementations by Zuul, and it is used in actuators' health check indicators, but it's not so much right?

Conclusion

We were able to integrate with Marathon. It's cool. Even more, we can even now get a list of services and their instances.

But we have at least two unsolved problems. First, our configuration contains only one instance of Marathon. If it fails, we wouldn't have information. No information, no right decisions. And second, we are not able to load balance at the moment without some additional explicit programming in every app that we develop. So, actually, we have a toy, not a useful and production-ready solution. 

See you in the next part!

Using Containers? Read our Kubernetes Comparison eBook to learn the positives and negatives of Kubernetes, Mesos, Docker Swarm and EC2 Container Services.

Topics:
spring cloud ,mesosphere marathon ,cloud ,service discovery

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}