Service Discovery With NGINX Plus and Consul

Service discovery is a key component of most distributed systems and service-oriented or microservices architectures. Because service instances have dynamically assigned network locations which can change over time due to autoscaling, failures, or upgrades, clients need a sophisticated service discovery mechanism for tracking the current network location of a service instance.

This blog post describes how to dynamically add or remove load-balanced servers that are registered with Consul, a popular tool for service discovery. The solution uses NGINX Plus’ on-the-fly reconfiguration API, which eliminates the need to reload NGINX Plus configuration file when changing the set of load-balanced servers. With NGINX Plus Release 8 (R8), changes you make with the API now persist across restarts and configuration reloads.

Consul, developed by HashiCorp, is most widely used for discovering and configuring services in your infrastructure. It also provides failure detection (health checking), a key/value store, and support for multiple data centers out of the box.

To make it easier to combine the NGINX Plus on-the-fly reconfiguration API with Consul, we’ve created a GitHub project called consul-demo, with step-by-step instructions for creating the configuration described this blog post. In this post, we will walk you through the proof of concept. Using tools like Docker, Docker Compose, Homebrew, and jq, you can spin up a Docker–based environment in which NGINX Plus load balances HTTP traffic to a couple of hello-world applications, with all components running in separate Docker containers.

How the Demo Works

First we spin up a separate Docker container for each of the following apps:

• Consul – Performs service discovery
• Registrator – Registers services with Consul. Registrator monitors for containers being started and stopped and updates Consul when a container changes state.
• tutum/hello-world – Simulates a service (a simple hello-world app)
• google/golang-hello – Simulates a second service (another simple hello-world app)
• NGINX Plus R8 – Load balances the other listed services

The NGINX Plus container listens on the public port 80, and the built-in NGINX Plus live activity monitoring dashboard on port 8080. The Consul container listens on ports 8300, 8400, 8500, and 53. It advertises a fixed IP address of 10.0.2.15 for intercontainer communication.

Registrator monitors Docker for new containers that are launched with exposed ports and registers the associated service with Consul. By setting environment variables within the containers, we can be more explicit about how to register the services with Consul. For each hello-world container, we set theSERVICE_TAGS environment variable to production to identify the container as an upstream server for load balancing by NGINX Plus. When a container quits or is removed, Registrator removes it from the Consul service catalog automatically.

Finally, we configure Consul watches in a JSON file included in consul-demo, so that every time there is an update to the list of registered services, an external handler (script.sh) is invoked. This bash script gets the list of all current NGINX Plus upstream servers, uses the Consul services API to loop through all the containers registered with Consul that are tagged production and uses NGINX Plus’ on-the-fly reconfiguration API (upstream_conf) to add them to the NGINX Plus upstream group if they’re not listed already. It also then removes from the NGINX Plus upstream group any production-tagged containers that are not registered with Consul.

Summary

Using a script like the one provided in our demo automates the process of upstream reconfiguration in NGINX Plus by dynamically reconfiguring your upstream groups based on the services registered with Consul. This automation also frees you from having to figure out how to issue the API calls correctly and reduces the amount of time between the state change of a service in Consul and its addition or removal within the NGINX Plus upstream group.