{{announcement.body}}
{{announcement.title}}

Spring Cloud Kubernetes For Hybrid Microservices Architecture

DZone 's Guide to

Spring Cloud Kubernetes For Hybrid Microservices Architecture

The only problem with starting applications outside Kubernetes is that there is no auto-configured registration mechanism — find out how to fix it!

· Microservices Zone ·
Free Resource

Spring (cloud) has sprung!

You might use Spring Cloud Kubernetes to build applications running both inside and outside the Kubernetes cluster. The only problem with starting applications outside Kubernetes is that there is no auto-configured registration mechanism.

Spring Cloud Kubernetes delegates registration to the platform, what is an obvious behavior if you are deploying your application internally using Kubernetes objects. With the external application, the situation is different. You should guarantee registration by yourself on the application side.

You may also like: Quick Guide to Microservices With Kubernetes, Spring Boot 2.0, and Docker

This article is an explanation of motivation to add auto-registration mechanisms to Spring Cloud Kubernetes project only for external applications. Let's consider the architecture where some microservices are running outside the Kubernetes cluster and some others are running inside it. There can be many explanations for such a situation. The most obvious explanation seems to be a migration of your microservices from older infrastructure to Kubernetes.

Assuming it is still in progress, you have some microservices already moved to the cluster, while some others still running on the older infrastructure. Moreover, you can decide to start some kind of experimental cluster with only a few of your applications, until you have more experience with using Kubernetes on production. I think it is not a very rare case.

Of course, there are different approaches to that issue. For example, you may maintain two independent microservices-based architectures, with different discovery registry and configuration sources. But you can also connect external microservices through Kubernetes API with the cluster to load configuration from ConfigMap or Secret, and register them there allow inter-service communication with Spring Cloud Kubernetes Ribbon.

The sample application source code is available on GitHub under branch hybrid in the sample-spring-microservices-Kubernetes repository: https://github.com/piomin/sample-spring-microservices-kubernetes/tree/hybrid.

Architecture

We move one of the sample microservices employee-service, described in the mentioned article, outside the Kubernetes cluster. Now, the applications which are communicating with employee-service need to use the addresses outside the cluster.

Also, they should be able to handle a port number dynamically generated on the application during startup (server.port=0). Our applications are still distributed across different namespaces, so it is important to enabling the multi-namespaces discovery features — also described in my previous article. The application employee-service is connecting to MongoDB, which is still deployed on Kubernetes. In that case, the integration is performed via Kubernetes Service. The following picture illustrates our current architecture.

kubernetes chart

Kubernetes PropertySource

The situation with distributed configuration is clear. We don't have to implement any additional code to be able to use it externally. Just, before starting client application we have to set the environment variable KUBERNETES_NAMESPACE. Since we set it to external we first need to create such a namespace.

Then we may apply some property sources to that namespace. The configuration is consisting of Kubernetes ConfigMap and Secret. We store there Mongo location, credentials, and some other properties. Here's our ConfigMap declaration.

Java




x
18


 
1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: employee
5
data:
6
  application.yaml: |-
7
    logging.pattern.console: "%d{HH:mm:ss} ${LOG_LEVEL_PATTERN:-%5p} %m%n"
8
    spring:
9
      cloud:
10
        kubernetes:
11
          discovery:
12
            all-namespaces: true
13
            register: true
14
      data:
15
        mongodb:
16
          database: admin
17
          host: 192.168.99.100
18
          port: 32612



The port number is taken from mongodb Service, which is deployed as NodePort type.

Here's our Secret.

Java




xxxxxxxxxx
1


 
1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: employee
5
type: Opaque
6
data:
7
  spring.data.mongodb.username: UGlvdF8xMjM=
8
  spring.data.mongodb.password: cGlvdHI=



Then, we are creating resources inside external namespace.

In bootstrap.yml file we need to set the address of Kubernetes API server and property responsible for trusting server's cert. We should also enable using Secret as a property source, which is disabled by default for Spring Cloud Kubernetes Config.

Java




xxxxxxxxxx
1
10


 
1
spring:
2
  application:
3
    name: employee
4
  cloud:
5
    kubernetes:
6
      secrets:
7
        enableApi: true
8
      client:
9
        masterUrl: 192.168.99.100:8443
10
        trustCerts: true



External Registration Implementation

The situation with service discovery is much more complicated. Since Spring Cloud Kubernetes delegates discovery to the platform, what is perfectly right for internal applications, the lack of auto-configured registration is a problem for external applications. That's why I decided to implement a module for Spring Cloud Kubernetes auto-configured registration for external application.

Currently, it is available inside our sample repository as the spring-cloud-Kubernetes-discovery-ext module. It is implemented according to the Spring Cloud Discovery registration pattern. Let's begin with dependencies. We just need to include spring-cloud-starter-Kubernetes, which contains core and discovery modules.

Java




xxxxxxxxxx
1


 
1
<dependency>
2
    <groupId>org.springframework.cloud</groupId>
3
    <artifactId>spring-cloud-starter-kubernetes</artifactId>
4
</dependency>



Here's our registration object. It implements Registration interface from Spring Cloud Commons, which defines some basic getters. We should provide the hostname, port, serviceId, etc.

Java




xxxxxxxxxx
1
75


 
1
public class KubernetesRegistration implements Registration {
2
 
3
    private KubernetesDiscoveryProperties properties;
4
 
5
    private String serviceId;
6
    private String instanceId;
7
    private String host;
8
    private int port;
9
    private Map<String, String> metadata = new HashMap<>();
10
 
11
    public KubernetesRegistration(KubernetesDiscoveryProperties properties) {
12
        this.properties = properties;
13
    }
14
 
15
    @Override
16
    public String getInstanceId() {
17
        return instanceId;
18
    }
19
 
20
    @Override
21
    public String getServiceId() {
22
        return serviceId;
23
    }
24
 
25
    @Override
26
    public String getHost() {
27
        return host;
28
    }
29
 
30
    @Override
31
    public int getPort() {
32
        return port;
33
    }
34
 
35
    @Override
36
    public boolean isSecure() {
37
        return false;
38
    }
39
 
40
    @Override
41
    public URI getUri() {
42
        return null;
43
    }
44
 
45
    @Override
46
    public Map<String, String> getMetadata() {
47
        return metadata;
48
    }
49
 
50
    @Override
51
    public String getScheme() {
52
        return "http";
53
    }
54
 
55
    public void setServiceId(String serviceId) {
56
        this.serviceId = serviceId;
57
    }
58
 
59
    public void setInstanceId(String instanceId) {
60
        this.instanceId = instanceId;
61
    }
62
 
63
    public void setHost(String host) {
64
        this.host = host;
65
    }
66
 
67
    public void setPort(int port) {
68
        this.port = port;
69
    }
70
 
71
    public void setMetadata(Map<String, String> metadata) {
72
        this.metadata = metadata;
73
    }
74
 
75
}



We have some additional configuration properties in comparison to Spring Cloud Kubernetes Discovery. They are available under the same prefix spring.cloud.kubernetes.discovery.

Java




xxxxxxxxxx
1
12


 
1
@ConfigurationProperties("spring.cloud.kubernetes.discovery")
2
public class KubernetesRegistrationProperties {
3
 
4
    private String ipAddress;
5
    private String hostname;
6
    private boolean preferIpAddress;
7
    private Integer port;
8
    private boolean register;
9
     
10
    // GETTERS AND SETTERS
11
     
12
}



There is also a class that should extend abstract AbstractAutoServiceRegistration. It is responsible for managing the registration process. First, it enables the registration mechanism only if the application is running outside Kubernetes.

It uses PodUtils bean defined in Spring Cloud Kubernetes Core for that. It also implements a method for building registration objects. The port may be generated dynamically on startup. The rest of the process is performed inside the abstract subclass.

Java




xxxxxxxxxx
1
61


 
1
public class KubernetesAutoServiceRegistration extends AbstractAutoServiceRegistration<KubernetesRegistration> {
2
 
3
    private KubernetesDiscoveryProperties properties;
4
    private KubernetesRegistrationProperties registrationProperties;
5
    private KubernetesRegistration registration;
6
    private PodUtils podUtils;
7
 
8
    KubernetesAutoServiceRegistration(ServiceRegistry<KubernetesRegistration> serviceRegistry,
9
                                      AutoServiceRegistrationProperties autoServiceRegistrationProperties,
10
                                      KubernetesRegistration registration, KubernetesDiscoveryProperties properties,
11
                                      KubernetesRegistrationProperties registrationProperties, PodUtils podUtils) {
12
        super(serviceRegistry, autoServiceRegistrationProperties);
13
        this.properties = properties;
14
        this.registrationProperties = registrationProperties;
15
        this.registration = registration;
16
        this.podUtils = podUtils;
17
    }
18
 
19
    public void setRegistration(int port) throws UnknownHostException {
20
        String ip = registrationProperties.getIpAddress() != null ? registrationProperties.getIpAddress() : InetAddress.getLocalHost().getHostAddress();
21
        registration.setHost(ip);
22
        registration.setPort(port);
23
        registration.setServiceId(getAppName(properties, getContext().getEnvironment()) + "." + getNamespace(getContext().getEnvironment()));
24
        registration.getMetadata().put("namespace", getNamespace(getContext().getEnvironment()));
25
        registration.getMetadata().put("name", getAppName(properties, getContext().getEnvironment()));
26
        this.registration = registration;
27
    }
28
 
29
    @Override
30
    protected Object getConfiguration() {
31
        return properties;
32
    }
33
 
34
    @Override
35
    protected boolean isEnabled() {
36
        return !podUtils.isInsideKubernetes();
37
    }
38
 
39
    @Override
40
    protected KubernetesRegistration getRegistration() {
41
        return registration;
42
    }
43
 
44
    @Override
45
    protected KubernetesRegistration getManagementRegistration() {
46
        return registration;
47
    }
48
 
49
    public String getAppName(KubernetesDiscoveryProperties properties, Environment env) {
50
        final String appName = properties.getServiceName();
51
        if (StringUtils.hasText(appName)) {
52
            return appName;
53
        }
54
        return env.getProperty("spring.application.name", "application");
55
    }
56
 
57
    public String getNamespace(Environment env) {
58
        return env.getProperty("KUBERNETES_NAMESPACE", "external");
59
    }
60
 
61
}



The process should be initialized just after the application startup. To catch startup event we prepare bean that implements SmartApplicationListener interface. The listener method calls bean KubernetesAutoServiceRegistration to prepare the registration object and start the process.

Java




xxxxxxxxxx
1
39


 
1
public class KubernetesAutoServiceRegistrationListener implements SmartApplicationListener {
2
 
3
    private static final Logger LOGGER = LoggerFactory.getLogger(KubernetesAutoServiceRegistrationListener.class);
4
 
5
    private final KubernetesAutoServiceRegistration autoServiceRegistration;
6
 
7
    KubernetesAutoServiceRegistrationListener(KubernetesAutoServiceRegistration autoServiceRegistration) {
8
        this.autoServiceRegistration = autoServiceRegistration;
9
    }
10
 
11
    @Override
12
    public boolean supportsEventType(Class<? extends ApplicationEvent> eventType) {
13
        return WebServerInitializedEvent.class.isAssignableFrom(eventType);
14
    }
15
 
16
    @Override
17
    public boolean supportsSourceType(Class<?> sourceType) {
18
        return true;
19
    }
20
 
21
    @Override
22
    public int getOrder() {
23
        return 0;
24
    }
25
 
26
    @Override
27
    public void onApplicationEvent(ApplicationEvent applicationEvent) {
28
        if (applicationEvent instanceof WebServerInitializedEvent) {
29
            WebServerInitializedEvent event = (WebServerInitializedEvent) applicationEvent;
30
            try {
31
                autoServiceRegistration.setRegistration(event.getWebServer().getPort());
32
                autoServiceRegistration.start();
33
            } catch (UnknownHostException e) {
34
                LOGGER.error("Error registering to kubernetes", e);
35
            }
36
        }
37
    }
38
 
39
}



Here's the auto-configuration for all previously described beans.

Java




xxxxxxxxxx
1
36


 
1
@Configuration
2
@ConditionalOnProperty(name = "spring.cloud.kubernetes.discovery.register", havingValue = "true")
3
@AutoConfigureAfter({AutoServiceRegistrationConfiguration.class, KubernetesServiceRegistryAutoConfiguration.class})
4
public class KubernetesAutoServiceRegistrationAutoConfiguration {
5
 
6
    @Autowired
7
    AutoServiceRegistrationProperties autoServiceRegistrationProperties;
8
 
9
    @Bean
10
    @ConditionalOnMissingBean
11
    public KubernetesAutoServiceRegistration autoServiceRegistration(
12
            @Qualifier("serviceRegistry") KubernetesServiceRegistry registry,
13
            AutoServiceRegistrationProperties autoServiceRegistrationProperties,
14
            KubernetesDiscoveryProperties properties,
15
            KubernetesRegistrationProperties registrationProperties,
16
            KubernetesRegistration registration, PodUtils podUtils) {
17
        return new KubernetesAutoServiceRegistration(registry,
18
                autoServiceRegistrationProperties, registration, properties, registrationProperties, podUtils);
19
    }
20
 
21
    @Bean
22
    public KubernetesAutoServiceRegistrationListener listener(KubernetesAutoServiceRegistration registration) {
23
        return new KubernetesAutoServiceRegistrationListener(registration);
24
    }
25
 
26
    @Bean
27
    public KubernetesRegistration registration(KubernetesDiscoveryProperties properties) throws UnknownHostException {
28
        return new KubernetesRegistration(properties);
29
    }
30
 
31
    @Bean
32
    public KubernetesRegistrationProperties kubernetesRegistrationProperties() {
33
        return new KubernetesRegistrationProperties();
34
    }
35
 
36
}



Finally, we may proceed to the most important step — an integration with Kubernetes API. Spring Cloud Kubernetes uses Fabric Kubernetes Client for communication with master API. The KubernetesClient bean is already auto-configured, so we may inject it.

The register and deregister methods are implemented in class KubernetesServiceRegistry that implements ServiceRegistry interface. Discovery in Kubernetes is configured via Endpoint API.

Each Endpoint contains a list of EndpointSubset that stores a list of registered IPs inside EndpointAddress and a list of listening ports inside EndpointPort. Here's the implementation of register and deregister methods.

Java




xxxxxxxxxx
1
78


 
1
public class KubernetesServiceRegistry implements ServiceRegistry<KubernetesRegistration> {
2
 
3
    private static final Logger LOG = LoggerFactory.getLogger(KubernetesServiceRegistry.class);
4
 
5
    private final KubernetesClient client;
6
    private KubernetesDiscoveryProperties properties;
7
 
8
    public KubernetesServiceRegistry(KubernetesClient client, KubernetesDiscoveryProperties properties) {
9
        this.client = client;
10
        this.properties = properties;
11
    }
12
 
13
    @Override
14
    public void register(KubernetesRegistration registration) {
15
        LOG.info("Registering service with kubernetes: " + registration.getServiceId());
16
        Resource<Endpoints, DoneableEndpoints> resource = client.endpoints()
17
                .inNamespace(registration.getMetadata().get("namespace"))
18
                .withName(registration.getMetadata().get("name"));
19
        Endpoints endpoints = resource.get();
20
        if (endpoints == null) {
21
            Endpoints e = client.endpoints().create(create(registration));
22
            LOG.info("New endpoint: {}",e);
23
        } else {
24
            try {
25
                Endpoints updatedEndpoints = resource.edit()
26
                        .editMatchingSubset(builder -> builder.hasMatchingPort(v -> v.getPort().equals(registration.getPort())))
27
                        .addToAddresses(new EndpointAddressBuilder().withIp(registration.getHost()).build())
28
                        .endSubset()
29
                        .done();
30
                LOG.info("Endpoint updated: {}", updatedEndpoints);
31
            } catch (RuntimeException e) {
32
                Endpoints updatedEndpoints = resource.edit()
33
                        .addNewSubset()
34
                        .withPorts(new EndpointPortBuilder().withPort(registration.getPort()).build())
35
                        .withAddresses(new EndpointAddressBuilder().withIp(registration.getHost()).build())
36
                        .endSubset()
37
                        .done();
38
                LOG.info("Endpoint updated: {}", updatedEndpoints);
39
            }
40
        }
41
 
42
    }
43
 
44
    @Override
45
    public void deregister(KubernetesRegistration registration) {
46
        LOG.info("De-registering service with kubernetes: " + registration.getInstanceId());
47
        Resource<Endpoints, DoneableEndpoints> resource = client.endpoints()
48
                .inNamespace(registration.getMetadata().get("namespace"))
49
                .withName(registration.getMetadata().get("name"));
50
 
51
        EndpointAddress address = new EndpointAddressBuilder().withIp(registration.getHost()).build();
52
        Endpoints updatedEndpoints = resource.edit()
53
                .editMatchingSubset(builder -> builder.hasMatchingPort(v -> v.getPort().equals(registration.getPort())))
54
                .removeFromAddresses(address)
55
                .endSubset()
56
                .done();
57
        LOG.info("Endpoint updated: {}", updatedEndpoints);
58
 
59
        resource.get().getSubsets().stream()
60
                .filter(subset -> subset.getAddresses().size() == 0)
61
                .forEach(subset -> resource.edit()
62
                        .removeFromSubsets(subset)
63
                        .done());
64
    }
65
 
66
    private Endpoints create(KubernetesRegistration registration) {
67
        EndpointAddress address = new EndpointAddressBuilder().withIp(registration.getHost()).build();
68
        EndpointPort port = new EndpointPortBuilder().withPort(registration.getPort()).build();
69
        EndpointSubset subset = new EndpointSubsetBuilder().withAddresses(address).withPorts(port).build();
70
        ObjectMeta metadata = new ObjectMetaBuilder()
71
                .withName(registration.getMetadata().get("name"))
72
                .withNamespace(registration.getMetadata().get("namespace"))
73
                .build();
74
        Endpoints endpoints = new EndpointsBuilder().withSubsets(subset).withMetadata(metadata).build();
75
        return endpoints;
76
    }
77
 
78
}



The auto-configuration beans are registered in spring.factories file.

Java




xxxxxxxxxx
1


 
1
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
2
org.springframework.cloud.kubernetes.discovery.ext.KubernetesServiceRegistryAutoConfiguration,\
3
org.springframework.cloud.kubernetes.discovery.ext.KubernetesAutoServiceRegistrationAutoConfiguration



Enabling Registration

Now, we may include already created library to any Spring Cloud application running outside Kubernetes, for example to the employee-service. We are using it together with Spring Cloud Kubernetes.

Java




xxxxxxxxxx
1


 
1
<dependency>
2
    <groupId>org.springframework.cloud</groupId>
3
    <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
4
</dependency>
5
<dependency>
6
    <groupId>pl.piomin.services</groupId>
7
    <artifactId>spring-cloud-kubernetes-discovery-ext</artifactId>
8
    <version>1.0-SNAPSHOT</version>
9
</dependency>



The registration is still disabled since we won't set properly spring.cloud.kubernetes.discovery.register to true.

Java




xxxxxxxxxx
1


 
1
spring:
2
  cloud:
3
    kubernetes:
4
      discovery:
5
        register: true



Sometimes it might be used to set the static IP address in configuration, in case you would have multiple network interfaces.

Java




xxxxxxxxxx
1


 
1
spring:
2
  cloud:
3
    kubernetes:
4
      discovery:
5
        ipAddress: 192.168.99.1



By setting 192.168.99.1 as a static IP address, I'm able to easily perform some tests with Minikube node, which is running on VM available under 192.168.99.100.

Manual Testing

Let's start employee-service locally. As you see on the screen below it has successfully load configuration from Kubernetes and connected with MongoDB running on the cluster.

After startup, the application has registered itself in Kubernetes.

We can view details of employee endpoint using kubectl describe endpoints command as shown below.

Finally, we can perform some test calls, for example via gateway-service running on Minikube.

Java




xxxxxxxxxx
1


 
1
$ curl http://192.168.99.100:31854/employee/actuator/info



Since Spring Cloud Kubernetes does not allow discovery across all namespaces for Ribbon clients, we should override Ribbon configuration using DiscoveryClient as shown below.

Java




xxxxxxxxxx
1
28


 
1
public class RibbonConfiguration {
2
 
3
    @Autowired
4
    private DiscoveryClient discoveryClient;
5
 
6
    private String serviceId = "client";
7
    protected static final String VALUE_NOT_SET = "__not__set__";
8
    protected static final String DEFAULT_NAMESPACE = "ribbon";
9
 
10
    public RibbonConfiguration () {
11
    }
12
 
13
    public RibbonConfiguration (String serviceId) {
14
        this.serviceId = serviceId;
15
    }
16
 
17
    @Bean
18
    @ConditionalOnMissingBean
19
    public ServerList<?> ribbonServerList(IClientConfig config) {
20
 
21
        Server[] servers = discoveryClient.getInstances(config.getClientName()).stream()
22
                .map(i -> new Server(i.getHost(), i.getPort()))
23
                .toArray(Server[]::new);
24
 
25
        return new StaticServerList(servers);
26
    }
27
 
28
}



Summary

There are some limitations related to discovery with Kubernetes. For example, there is not a build-in heartbeat mechanism, so we should take care of removing application endpoints on shutdown. Also, I'm not considering security aspects related to allowing discovery across all namespaces and allowing access to API for external applications.

I'm assuming you have guaranteed the required level of security when building your Kubernetes cluster, especially if you decide to allow external access to the API. API is still just API and we may use it. This article shows an example of a use case, which may be useful for you.

If you compare it with my previous article about Spring Cloud Kubernetes you see that with small configuration changes you can move application outside cluster without adding any new components for discovery or a distributed configuration.


Further Reading

Developing A Spring Boot Application for Kubernetes Cluster: A Tutorial [Part 1]

Spring Boot Microservice Development on Kubernetes [Videos]

Building Elastic Microservices With Kubernetes and Spring Boot From the Ground Up

Topics:
microservices ,kubernetes ,spring boot ,hybrid microservices ,microservices architecture

Published at DZone with permission of Piotr Mińkowski , DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}