Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Automating Nginx Plus High Availability With Chef

DZone's Guide to

Automating Nginx Plus High Availability With Chef

Learn how to automate NGINX Plus for high availability with Chef, from modifying the NGINX cookbook to prepping nodes for installation.

Free Resource

Transform incident management with machine learning and analytics to help you maintain optimal performance and availability while keeping pace with the growing demands of digital business with this eBook, brought to you in partnership with BMC.

In a previous blog post, we went over how to deploy NGINX Plus using Chef. In this blog post, we will build on the Chef configs from the last blog post and use Chef to deploy a highly available (HA) NGINX Plus active/passive cluster.

This post assumes that you have already run through the previous post and have a working Chef installation. If not, please review at least the first two sections in the previous post, Preparing Your Chef Environment and Downloading and Configuring the NGINX Cookbook.

To set up the HA active/passive cluster, we’re using the HA solution provided by NGINX, Inc., which is based on keepalived and the Virtual Router Redundancy Protocol (VRRP). The solution comes with an interactive script for creating the keepalived configuration file, but here we’re using Chef to automate the complete setup process.

Modifying the NGINX Cookbook

First, we set up the Chef files for installing of the NGINX Plus HA package (nginx-ha-keepalived) and creating the keepalived configuration file, keepalive.conf.

  1. Modify the existing plus_package recipe to include package and configuration templates for the HA solution, by adding the following code to the bottom of the plus_package.rb file (per the instructions in the previous post, the file is in the ~/chef-zero/playground/cookbooks/nginx/recipes directory).We are using the eth1 interface on each NGINX host, which makes the code a bit more complicated than if we used eth0. In case you are using eth0, the relevant code appears near the top of the file, commented out.This code does three things:
    • It looks up the IP address of the eth1 interface on the node where NGINX Plus is being installed and assigns the value to the origip variable so it can be passed to the template.
    • It finds the other node in the HA pair by using Chef’s search function to iterate through all Chef nodes, then looks up the IP address for that node’s eth1 interface and assigns the address to the ha_pair_ips variable.
    • It installs the nginx-ha-keepalived package, registers the keepalived service with Chef, and generates the keepalived.conf configuration file as a template, passing in the values of the origip and ha_pair_ips variables.
  2. 
    
    if node['nginx']['enable_ha_mode'] == 'true'
    
      ha_pair_ips = Array.new
    
      origip = "#{node[:network][:interfaces][:eth1][:addresses].detect{|k,v| v[:family] == 'inet'}.first}"
    
      # The code for finding the IP address of the eth0 interface
    
      # follows, commented out.
    
      #origip = "#{node[:ipaddress]}"
    
      #search(:node, "role:nginx_plus_ha") do |nodes|
    
      #  ha_pair_ips  true, :restart => true, :reload => true
    
        action   :enable
    
      end
    
      template '/etc/keepalived/keepalived.conf' do
    
        source 'nginx_plus_keepalived.conf.erb'
    
        owner 'root'
    
        group node['root_group']
    
        mode '0644'
    
        variables(
    
                  :myip => origip,
    
                  :ha_pair_ip => ha_pair_ips
    
                 )
    
        notifies :reload, 'service[keepalived]', :delayed
    
      end
    
    end
    You can download the full recipe file from the NGINX, Inc. website.
  3. Create the Chef template for creating keepalived.conf, by copying the following content to a new template file, nginx_plus_keepalived.conf.erb, in the ~/chef-zero/playground/cookbooks/nginx/templates/default directory.We’re using a combination of variables and attributes to pass the necessary information to keepalived.conf. We’ll set the attributes in the next step. Here we set the two variables in the template file to the host IP addresses that were set with the variables directive in the plus_package.rb recipe (modified in the previous step):
    • myip – The primary IP address used by keepalived to communicate with the other HA nodes. Corresponds to the origip variable in the plus_package recipe.
    • ha_pair_ip – An array containing the IP address of each host that has the nginx_plus_ha role in its run list; it is used to set the IP address of the peer (secondary) host in keepalived.conf. Corresponds to the ha_pair_ips array in the plus_package recipe.
  4. vrrp_script chk_nginx_service {
    
         script "/usr/lib/keepalived/nginx-ha-check"
    
         interval 3
    
         weight 50
    
    }
    vrrp_instance VI_1 {
    
     interface eth1
    
    
    
     state MASTER
    
     priority 151
    
    
    
    
    
     state BACKUP
    
     priority 150
    
    
    
     virtual_router_id 51
    
     advert_int 1
    
     unicast_src_ip 
    
     unicast_peer {
    
    
    
    
    
    
    
    
    
    
    
     }
    
     authentication {
    
     auth_type PASS
    
     auth_pass 
    
     }
    
     virtual_ipaddress {
    
    
    
     }
    
     track_script {
    
     chk_nginx_service
    
     }
    
     notify "/usr/lib/keepalived/nginx-ha-notify"
    
    }
  5. Create a role that sets attributes used in the recipe and template files created in the previous steps, by copying the following contents to a new role file, nginx_plus_ha.rb in the ~/chef-zero/playground/roles directory.Four attributes need to be set, and in the role we set the following three:
    • [nginx][ha_keepalived_key] – The authentication key used by keepalived to encrypt communication with the other HA nodes. Used in the template file.
    • [nginx][ha_vip] – The virtual IP address (VIP) advertised to clients as the address for NGINX Plus. The keepalived process assigns it to the primary NGINX Plus instance and is is responsible for transferring it to the other instance in case of failover. Used in the template file.
    • [nginx][enable_ha_mode] – Triggers the HA section of the plus_package recipe when set to true. Used in the recipe file.
  6. We don’t set the fourth attribute, ha_primary, in the role because it must be set on a per-host basis. It is used in the template file to set the state and priority values in keepalived.conf; those values in turn determine which NGINX Plus instance is marked as primary. If ha_primary is true for a node, it becomes the primary NGINX Plus instance in the HA pair; it becomes the secondary instance if the value is false.It is also possible to set the primary instance based on hostname matching instead of the ha_primary attribute, as we’ll cover in Setting the Primary Node Based on Hostname Matching. Also, you can set all four attributes on a per-host basis, but I prefer to keep as many of the shared attributes in a single location as possible.The run_list directive in the role references the nginx_plus role we created in the previous blog post, to save us from having to duplicate everything already defined in that role.
    name "nginx_plus_ha"
    
    description "An example role to install nginx plus in an HA cluster"
    
    run_list "role[nginx_plus]"
    
    default_attributes "nginx" => { "enable_ha_mode" => "true",
    
                                    "ha_keepalived_key" => "a0cf476cf069ea3dfa8940ff6d6bd885",
    
                                    "ha_vip" => "10.100.10.50"
    
                                  }

Preparing Nodes for Installation

Now we bootstrap the nodes and get them ready for the installation. Note that the longer outputs have been truncated, leaving only the most important output.

Setting Up the First Node

  1. Upload all of the files and bootstrap the first node:
    root@chef-server:~# cd chef-zero/playground/cookbooks/
    
    root@chef-server:~/chef-zero/playground/cookbooks# knife cookbook upload *
    
    Uploading apache2      [1.0.0]
    
    Uploading apt          [2.8.2]
    
    ...
    
    Uploading nginx        [2.7.6]
    
    ...
    
    Uploading yum          [3.8.1]
    
    Uploading yum-epel     [0.6.3]
    
    Uploaded 18 cookbooks.
    root@chef-server:~/chef-zero/playground/cookbooks# cd ../
    root@chef-server:~/chef-zero/playground# knife role from file roles/nginx_plus.rb
    
    Updated Role nginx_plus!
    root@chef-server:~/chef-zero/playground# knife role from file roles/nginx_plus_ha.rb
    
    Updated Role nginx_plus_ha!
    root@chef-server:~/chef-zero/playground# knife bootstrap -N chef-test-1 -x username --sudo 10.100.10.100
    
    Creating new client for chef-test-1
    
    Creating new node for chef-test-1
    
    Connecting to 10.100.10.100
    
    username@10.100.10.100's password:
    
    10.100.10.100 knife sudo password:
    
    Enter your password:
    
    10.100.10.100
    
    10.100.10.100 -----> Existing Chef installation detected
    
    10.100.10.100 Starting first Chef Client run...
    
    10.100.10.100 Starting Chef Client, version 12.6.0
    
    10.100.10.100 resolving cookbooks for run list: []
    
    10.100.10.100 Synchronizing Cookbooks:
    
    10.100.10.100 Compiling Cookbooks...
    
    10.100.10.100 [2016-02-07T06:17:13-08:00] WARN: Node chef-test-1 has an empty run list.
    
    10.100.10.100 Converging 0 resources
    
    10.100.10.100
    
    10.100.10.100 Running handlers:
    
    10.100.10.100 Running handlers complete
    
    10.100.10.100 Chef Client finished, 0/0 resources updated in 01 seconds
  2. Create a local copy of the node definition file, which we’ll edit as appropriate for the node we bootstrapped in the previous step, chef-test-1:
    root@chef-server:~/chef-zero/playground# knife node show chef-test-1 --format json > nodes/chef-test-1.json
  3. Edit chef-test-1.json to have the following contents. In particular, we’re updating the run list and setting the ha_primary attribute, as required for the HA deployment.
    {
    
      "name": "chef-test-1",
    
      "chef_environment": "_default",
    
      "run_list": [
    
                 "role[nginx_plus_ha]"
    
      ]
    
    ,
    
      "normal": {
    
        "nginx": {
    
          "ha_primary": "true"
    
        },
    
        "tags": [
    ]
    
     }
    
    }
  4. Push the changed node definition to Chef:
    root@chef-server:~/chef-zero/playground# knife node from file nodes/chef-test-1.json
    
    Updated Node chef-test-1!
  5. Log in on the chef-test-1 node and run the chef‑client command to get everything configured:
    username@chef-test-1:~$ sudo chef-client
    
    Starting Chef Client, version 12.6.0
    
    resolving cookbooks for run list: ["nginx"]
    
    Synchronizing Cookbooks:
    
      - ohai (2.0.1)
    
      - build-essential (2.2.4)
    
      - nginx (2.7.6)
    
      - yum-epel (0.6.3)
    
      - apt (2.8.2)
    
      - bluepill (2.4.0)
    
      - runit (1.7.2)
    
      - rsyslog (2.1.0)
    
      - packagecloud (0.1.0)
    
      - yum (3.8.1)
    
    Compiling Cookbooks...
    
    ….
    
      * template[/etc/keepalived/keepalived.conf] action create
    
        - create new file /etc/keepalived/keepalived.conf
    
        - update content in file /etc/keepalived/keepalived.conf from none to 9816fd
    
        --- /etc/keepalived/keepalived.conf    2016-02-07 06:34:18.117013010 -0800
    
        +++ /etc/keepalived/.keepalived.conf20160207-1079-sjg8xn    2016-02-07 06:34:18.117013010 -0800
    
        @@ -1 +1,28 @@
    
        +vrrp_script chk_nginx_service {
    
        +    script "/usr/lib/keepalived/nginx-ha-check"
    
        +    interval 3
    
        +    weight 50
    
        +}
    
        +
    
        +vrrp_instance VI_1 {
    
        +    interface eth1
    
        +    state MASTER
    
        +    priority 151
    
        +    virtual_router_id 51
    
        +    advert_int 1
    
        +    unicast_src_ip 10.100.10.100
    
        +    unicast_peer {
    
        +    }
    
        +    authentication {
    
        +        auth_type PASS
    
        +        auth_pass a0cf476cf069ea3dfa8940ff6d6bd885
    
        +    }
    
        +    virtual_ipaddress {
    
        +        10.100.10.50
    
        +    }
    
        +    track_script {
    
        +        chk_nginx_service
    
        +    }
    
        +    notify "/usr/lib/keepalived/nginx-ha-notify"
    
        +}
    
        - change mode from '' to '0644'
    
        - change owner from '' to 'root'
    
        - change group from '' to 'root'
    
    Recipe: nginx::default
    
      * service[nginx] action start (up to date)
    
      * service[nginx] action reload
    
        - reload service service[nginx]
    
    Recipe: nginx::plus_package
    
      * service[keepalived] action reload (up to date)
    Running handlers:
    
    Running handlers complete
    
    Chef Client finished, 18/50 resources updated in 07 seconds

If we look at keepalived.conf at this point, we see the template hasn’t set any values in the unicast_peer section, because we’ve registered just this one node with Chef. The following command shows that the VIP specified in the role file, 10.100.10.50, is assigned to eth1 on this node, making it the primary HA node:

username@chef-test-1:~$ ip addr show eth1

3: eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:c8:66:ee brd ff:ff:ff:ff:ff:ff

    inet 10.100.10.100/24 brd 10.100.10.255 scope global eth1

       valid_lft forever preferred_lft forever

    inet 10.100.10.50/32 scope global eth1

       valid_lft forever preferred_lft forever

    inet6 fe80::a00:27ff:fec8:66ee/64 scope link

       valid_lft forever preferred_lft forever

Setting Up the Second Node

Now, let’s get the second node bootstrapped and make sure all of the values are being propagated properly. We know from the first node that the run list works as expected, so we can combine some steps and set the run list and the ha_primary attribute directly with the knife bootstrap command, which we run on the Chef server.

root@chef-server:~/chef-zero/playground# knife bootstrap -N chef-test-2 -x username --json-attributes "{\"nginx\": {\"ha_primary\": \"false\"}}" --sudo --run-list "role[nginx_plus_ha]" 10.100.10.102

Creating new client for chef-test-2

Creating new node for chef-test-2

Connecting to 10.100.10.102

username@10.100.10.102's password:

10.100.10.102 sudo: unable to resolve host chef-test

10.100.10.102 knife sudo password:

Enter your password:

10.100.10.102

10.100.10.102 -----> Existing Chef installation detected

10.100.10.102 Starting first Chef Client run...

10.100.10.102 Starting Chef Client, version 12.6.0

10.100.10.102 resolving cookbooks for run list: ["nginx"]

10.100.10.102 Synchronizing Cookbooks:

10.100.10.102   - bluepill (2.4.0)

10.100.10.102   - apt (2.8.2)

10.100.10.102   - build-essential (2.2.4)

10.100.10.102   - ohai (2.0.1)

10.100.10.102   - nginx (2.7.6)

10.100.10.102   - runit (1.7.2)

10.100.10.102   - yum-epel (0.6.3)

10.100.10.102   - rsyslog (2.1.0)

10.100.10.102   - packagecloud (0.1.0)

10.100.10.102   - yum (3.8.1)

10.100.10.102 Compiling Cookbooks...

…

10.100.10.102   * service[keepalived] action enable (up to date)

10.100.10.102   * template[/etc/keepalived/keepalived.conf] action create

10.100.10.102     - create new file /etc/keepalived/keepalived.conf

10.100.10.102     - update content in file /etc/keepalived/keepalived.conf from none to a9363c

10.100.10.102     --- /etc/keepalived/keepalived.conf    2016-02-07 06:45:10.529976825 -0800

10.100.10.102     +++ /etc/keepalived/.keepalived.conf20160207-11317-1env6hu    2016-02-07 06:45:10.529976825 -0800

10.100.10.102     @@ -1 +1,29 @@

10.100.10.102     +vrrp_script chk_nginx_service {

10.100.10.102     +    script "/usr/lib/keepalived/nginx-ha-check"

10.100.10.102     +    interval 3

10.100.10.102     +    weight 50

10.100.10.102     +}

10.100.10.102     +

10.100.10.102     +vrrp_instance VI_1 {

10.100.10.102     +    interface eth1

10.100.10.102     +    state BACKUP

10.100.10.102     +    priority 150

10.100.10.102     +    virtual_router_id 51

10.100.10.102     +    advert_int 1

10.100.10.102     +    unicast_src_ip 10.100.10.102

10.100.10.102     +    unicast_peer {

10.100.10.102     +          10.100.10.100

10.100.10.102     +    }

10.100.10.102     +    authentication {

10.100.10.102     +        auth_type PASS

10.100.10.102     +        auth_pass a0cf476cf069ea3dfa8940ff6d6bd885

10.100.10.102     +    }

10.100.10.102     +    virtual_ipaddress {

10.100.10.102     +        10.100.10.50

10.100.10.102     +    }

10.100.10.102     +    track_script {

10.100.10.102     +        chk_nginx_service

10.100.10.102     +    }

10.100.10.102     +    notify "/usr/lib/keepalived/nginx-ha-notify"

10.100.10.102     +}

10.100.10.102     - change mode from '' to '0644'

10.100.10.102     - change owner from '' to 'root'

10.100.10.102     - change group from '' to 'root'

10.100.10.102 Recipe: nginx::default

10.100.10.102   * service[nginx] action start (up to date)

10.100.10.102   * service[nginx] action reload

10.100.10.102     - reload service service[nginx]

10.100.10.102 Recipe: nginx::plus_package

10.100.10.102   * service[keepalived] action reload

10.100.10.102     - reload service service[keepalived]

10.100.10.102

10.100.10.102 Running handlers:

10.100.10.102 Running handlers complete

10.100.10.102 Chef Client finished, 18/50 resources updated in 10 seconds

If we look at keepalived.conf at this point, we see that there is a peer set in the unicast_peer section. But the following command shows that chef‑test‑2, which we intend to be the secondary node, is also assigned the VIP (10.100.10.50). This is because we haven’t yet updated the Chef configuration on chef‑test‑1 to make its keepalived aware of the secondary node.

username@chef-test-2:~$ ip addr show eth1

3: eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:6d:d9:64 brd ff:ff:ff:ff:ff:ff

    inet 10.100.10.102/24 brd 10.100.10.255 scope global eth1

       valid_lft forever preferred_lft forever

    inet 10.100.10.50/32 scope global eth1

       valid_lft forever preferred_lft forever

    inet6 fe80::a00:27ff:fe6d:d964/64 scope link

       valid_lft forever preferred_lft forever

Synchronizing the Nodes

To make keepalived on chef-test-1 aware of chef-test-2 and its IP address, we rerun the chef‑client command on chef‑test‑1:

username@chef-test-1:~$ sudo chef-client

Starting Chef Client, version 12.6.0

resolving cookbooks for run list: ["nginx"]

Synchronizing Cookbooks:

  - ohai (2.0.1)

  - build-essential (2.2.4)

  - nginx (2.7.6)

  - yum-epel (0.6.3)

  - apt (2.8.2)

  - bluepill (2.4.0)

  - runit (1.7.2)

  - rsyslog (2.1.0)

  - packagecloud (0.1.0)

  - yum (3.8.1)

Compiling Cookbooks...

….

  * template[/etc/keepalived/keepalived.conf] action create

    - update content in file /etc/keepalived/keepalived.conf from 9816fd to af7ce0

    --- /etc/keepalived/keepalived.conf    2016-02-07 06:34:18.117013010 -0800

    +++ /etc/keepalived/.keepalived.conf20160207-3369-73qgm3    2016-02-07 06:53:04.593013010 -0800

    @@ -12,6 +12,7 @@

         advert_int 1

         unicast_src_ip 10.100.10.100

         unicast_peer {

    +          10.100.10.102

         }

         authentication {

             auth_type PASS

Recipe: nginx::default

  * service[nginx] action start (up to date)

  * service[nginx] action reload

    - reload service service[nginx]

Recipe: nginx::plus_package

  * service[keepalived] action reload (up to date)
Running handlers:

Running handlers complete

Chef Client finished, 2/47 resources updated in 05 seconds

We see that chef-test-1 is still assigned the VIP:

username@chef-test-1:~$ ip addr show eth1

3: eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:c8:66:ee brd ff:ff:ff:ff:ff:ff

    inet 10.100.10.100/24 brd 10.100.10.255 scope global eth1

       valid_lft forever preferred_lft forever

    inet 10.100.10.50/32 scope global eth1

       valid_lft forever preferred_lft forever

    inet6 fe80::a00:27ff:fec8:66ee/64 scope link

       valid_lft forever preferred_lft forever

And chef-test-2, as the secondary node, is now assigned only its physical IP address:

username@chef-test-2:~$ ip addr show eth1

3: eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

    link/ether 08:00:27:6d:d9:64 brd ff:ff:ff:ff:ff:ff

    inet 10.100.10.102/24 brd 10.100.10.255 scope global eth1

       valid_lft forever preferred_lft forever

    inet6 fe80::a00:27ff:fe6d:d964/64 scope link

       valid_lft forever preferred_lft forever

If you want to test the keepalived failover at this point, stop the NGINX Plus service on the primary node.

Setting the Primary Node Based on Hostname Matching

You can automate your HA setup even further if you indicate a host’s usual role (primary or secondary) in its hostname. Then, instead of setting the ha_primary attribute in the node definition file (as in Step 3 of Setting Up the First Node), you can have Chef set the attribute based on the hostname. This takes advantage of Chef’s flexibility in allowing you to set variables inside of recipes, which you do by using slightly modified Ruby code to inspect any of the attributes and membership information the Chef server is aware of. One example is Step 1 of Modifying the NGINX Cookbook, where we added code to the plus_package recipe to look up eth1‘s IP address.

Here we extend that code further by adding an if statement that inspects the hostname and sets ha_primary appropriately. It relies on the presence of the word primary or standby in the hostname. Put it after the code in plus_package.rb that assigns a value to the origip variable (the complete code appears in Step 1 of Modifying the NGINX Cookbook):

   ...
origip = "#{node[:network][:interfaces][:eth1][:addresses].detect{|k,v| v[:family] == 'inet'}.first}"

 search(:node, "role:nginx_plus_ha AND enable_ha_mode:#{node.nginx.enable_ha_mode} NOT name:#{node.name}") do |nodes|

 nodes["network"]["interfaces"]["eth1"]["addresses"].each_pair do |address,value|

 ha_pair_ips if node['name'].include? "primary"

 ha_primary = "true"

 elsif node['name'].include? "standby"

 ha_primary = "false"

 end
package 'nginx-ha-keepalived' do

 action :install

 end
...

Of course, you can match on terms in the hostname other than primary and standby, such as lb1 and lb2. You can also put this logic in the nginx_plus_keepalived.conf.erb template file instead, but that is less portable.

Summary

We have covered some more complex examples of what is possible using the Chef search function, as well as Ruby code in the cookbook recipes. It is possible to apply these techniques in many ways, using attributes and the search function to build very portable and flexible Chef configurations. We will cover some of these concepts in future blog posts.

Evolve your approach to Application Performance Monitoring by adopting five best practices that are outlined and explored in this e-book, brought to you in partnership with BMC.

Topics:
nginx ,automate ,performance ,chef

Published at DZone with permission of Patrick Nommensen, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}