{{announcement.body}}
{{announcement.title}}

Creating Dynamic VMs Infrastructure With Xen Hosts

DZone 's Guide to

Creating Dynamic VMs Infrastructure With Xen Hosts

In this article, take a look at creating dynamic VMs infrastructure with Xen hosts.

· DevOps Zone ·
Free Resource

Introduction

In today's world, the server infrastructure machines are either in on-premise data centers, private data centers, or public cloud data centers. These machines are either physical bare-metal machines, virtual machines (VMs) with hypervisors, or small containers like Docker containers on top of physical or virtual machines. These machines physically might be in the local lab. In a private data center scenario, where your own procured hosts are placed in a physical space that is shared in a third-party data center and connected remotely. Whereas in the public data centers like AWS, GCP, Azure, OCI the machines are either reserved or created on-demand for the highly scalable needs connecting remotely. Each of these have its own advantages w.r.t scalability, security, reliability, management, and costs associated with those infrastructures. 

The product development environment teams might need many servers during the SDLC process. Let us say if one had chosen the private data center with their own physical machines along with Xen Servers. Now, the challenge is how the VMs lifecycle is managed for provision or termination in similar to cloud environments with lean and agile processes.

This document is aiming to provide the basic infrastructure model, architecture, minimum APIs and sample code snippets so that one can easily build dynamic infrastructure environments.

Benefits

First, let's understand the typical sequence of the steps followed in this server's infrastructure process. You can recollect it as below.

  1. Procurement of the new machines by IT
  2. Host virtualization - Install Xen Server and Create VM Templates by IT
  3. Static VMs request by Dev and test teams via (say JIRA) tickets to IT
  4. Maintain the received VM IPs in a database or a static file or hardcoded in configuration files or CI tools like in Jenkins config.xml
  5. Monitor the VMs for health checks to make sure these are healthy before using to install the products
  6. Cleanup or uninstall before or after Server installations
  7. Windows might need some registry cleanup before installing the product
  8. Fixed allocation of VMs to an area or a team or dedicated to an engineer might have been done

Now, how can you make this process more lean and agile? Can you eliminate most of the above steps with simple automation?

Yes. In our environment, we had more than 1000 VMs and tried to achieved and mainly the below. 

"Disposable VMs on-demand as required during tests execution. Solve Windows cleanup issues with regular test cycles."

As you see below, using the dynamic VMs server manager API service, 6 out of 8 steps can be eliminated and it gives the unlimited infrastructure view for the entire product team. Only the first 2 steps - procure and host virtualization are needed. In effect, this saves in terms of time and cost!

[caption id="attachment_8597" align="alignnone" width="3218"]Typical flow to get infrastructure

Dynamic Infrastructure Model

The below picture shows our proposed infrastructure for a typical server product environments where 80% of docker containers, 15% as dynamic VMs, and 5% as static pooled VMs for special cases. This distribution can be adjusted based on what works most for your environment.

[caption id="attachment_8598" align="alignnone" width="3078"]Infrastructure model

From here on, we will discuss more about Dynamic VM server manager part.

Dynamic Server Manager architecture

In the dynamic VMs server manager, a simple API service where the below REST APIs can be exposed and can be used anywhere in the automated process. As the tech stack shows, python 3 and Python-based Xen APIs are used for actual creation of VMs with XenServer host. Flask is being used for the REST service layer creation. The OS can be any of your product supported platforms like windows2016, centos7, centos8, debian10, ubuntu18, oel8, suse15.

[caption id="attachment_8599" align="alignnone" width="3056"]Dynamic VMs server manager architecture

Save the history of the VMs to track the usage and time to provision or termination can be analyzed further. For storing the JSON document, Couchbase enterprise server, which is a NoSQL document database can be used.

Simple REST APIs

Method URI(s) Purpose
GET /showall Lists all VMs in JSON format
GET /getavailablecount/<os> Gets the list of available VMs count for the given <os>
GET

/getservers/<name>?os=<os>

/getservers/<name>?os=<os>&count=<count>

/getservers/<name>?os=<os>&count=<count>&cpus=<cpus>&mem=<memsize>

/getservers/<name>?os=<os>&expiresin=<minutes>

Provisions given <count>  VMs of <os>.

cpus count and mem size also can be supported.

expiresin parameter in minutes to get expiry (auto termination) of the VMs.

GET

/releaseservers/<name>?os=<os>

/releaseservers/<name>?os=<os>&count=<count>

Terminates given <count>  VMs of <os>

Pre-Requirements for Dynamic VM Targeted Xen Hosts

  • Identify targeted dynamic VM Xen Hosts
  • Copy/create the VM templates 
  • Move these Xen Hosts a separate VLAN/Subnet (work with IT) for IPs recycle

Implementation

At a high level -

  1. Create functions each REST API
  2. Call a common service to perform different REST actions.
  3. Understand the Xen Session creation, getting the records, cloning VM from template, attaching the right disk, waiting for the VM creation and IP received; deletion of VMs, deletion of disks
  4. Start a thread for expiry of VMs automatically
  5. Read the common configuration such as .ini format
  6. Understand working with Couchbase database and save documents
  7. Test all APIs with required OSes and parameters
  8. Fix issues if any
  9. Perform a POC with few Xen Hosts

The below code snippets can help you to understand even better.

APIs creation

Java
 




x
116


 
1
@app.route('/showall/<string:os>')
2
@app.route("/showall")
3
def showall_service(os=None):
4
    count, _ = get_all_xen_hosts_count(os)
5
    log.info("--> count: {}".format(count))
6
    all_vms = {}
7
    for xen_host_ref in range(1, count + 1):
8
        log.info("Getting xen_host_ref=" + str(xen_host_ref))
9
        all_vms[xen_host_ref] = perform_service(xen_host_ref, service_name='listvms', os=os)
10
    return json.dumps(all_vms, indent=2, sort_keys=True)
11
 
          
12
 
          
13
@app.route('/getavailablecount/<string:os>')
14
@app.route('/getavailablecount')
15
def getavailable_count_service(os='centos'):
16
    """
17
    Calculate the available count:
18
        Get Total CPUs, Total Memory
19
        Get Free CPUs, Free Memory
20
        Get all the VMs - CPUs and Memory allocated
21
        Get each OS template - CPUs and Memory
22
        Available count1 = (Free CPUs - VMs CPUs)/OS_Template_CPUs
23
        Available count2 = (Free Memory - VMs Memory)/OS_Template_Memory
24
        Return min(count1,count2)
25
    """
26
    count, available_counts, xen_hosts = get_all_available_count(os)
27
    log.info("{},{},{},{}".format(count, available_counts, xen_hosts, reserved_count))
28
    if count > reserved_count:
29
        count -= reserved_count
30
    log.info("Less reserved count: {},{},{},{}".format(count, available_counts, xen_hosts,
31
                                                 reserved_count))
32
    return str(count)
33
 
          
34
 
          
35
# /getservers/username?count=number&os=centos&ver=6&expiresin=30
36
@app.route('/getservers/<string:username>')
37
def getservers_service(username):
38
    global reserved_count
39
    if request.args.get('count'):
40
        vm_count = int(request.args.get('count'))
41
    else:
42
        vm_count = 1
43
    os_name = request.args.get('os')
44
    if request.args.get('cpus'):
45
        cpus_count = request.args.get('cpus')
46
    else:
47
        cpus_count = "default"
48
 
          
49
    if request.args.get('mem'):
50
        mem = request.args.get('mem')
51
    else:
52
        mem = "default"
53
 
          
54
    if request.args.get('expiresin'):
55
        exp = int(request.args.get('expiresin'))
56
    else:
57
        exp = MAX_EXPIRY_MINUTES
58
 
          
59
    if request.args.get('format'):
60
        output_format = request.args.get('format')
61
    else:
62
        output_format = "servermanager"
63
 
          
64
    xhostref = None
65
    if request.args.get('xhostref'):
66
        xhostref = request.args.get('xhostref')
67
    reserved_count += vm_count
68
    if xhostref:
69
        log.info("-->  VMs on given xenhost" + xhostref)
70
        vms_ips_list = perform_service(xhostref, 'createvm', os_name, username, vm_count,
71
                                       cpus=cpus_count, maxmemory=mem, expiry_minutes=exp,
72
                                       output_format=output_format)
73
        return json.dumps(vms_ips_list)
74
 
          
75
 ...
76
 
          
77
 
          
78
# /releaseservers/{username}
79
@app.route('/releaseservers/<string:username>/<string:available>')
80
@app.route('/releaseservers/<string:username>')
81
def releaseservers_service(username):
82
    if request.args.get('count'):
83
        vm_count = int(request.args.get('count'))
84
    else:
85
        vm_count = 1
86
    os_name = request.args.get('os')
87
    delete_vms_res = []
88
    for vm_index in range(vm_count):
89
        if vm_count > 1:
90
            vm_name = username + str(vm_index + 1)
91
        else:
92
            vm_name = username
93
        xen_host_ref = get_vm_existed_xenhost_ref(vm_name, 1, None)
94
        log.info("VM to be deleted from xhost_ref=" + str(xen_host_ref))
95
        if xen_host_ref != 0:
96
            delete_per_xen_res = perform_service(xen_host_ref, 'deletevm', os_name, vm_name, 1)
97
            for deleted_vm_res in delete_per_xen_res:
98
                delete_vms_res.append(deleted_vm_res)
99
    if len(delete_vms_res) < 1:
100
        return "Error: VM " + username + " doesn't exist"
101
    else:
102
        return json.dumps(delete_vms_res, indent=2, sort_keys=True)
103
 
          
104
 
          
105
def perform_service(xen_host_ref=1, service_name='list_vms', os="centos", vm_prefix_names="",
106
                    number_of_vms=1, cpus="default", maxmemory="default",
107
                    expiry_minutes=MAX_EXPIRY_MINUTES, output_format="servermanager",
108
                    start_suffix=0):
109
    xen_host = get_xen_host(xen_host_ref, os)
110
...
111
 
          
112
def main():
113
    # options = parse_arguments()
114
    set_log_level()
115
    app.run(host='0.0.0.0', debug=True)
116
 
          


 

Creation Xen session

Java
 




xxxxxxxxxx
1
14


 
1
def get_xen_session(xen_host_ref=1, os="centos"):
2
    xen_host = get_xen_host(xen_host_ref, os)
3
    if not xen_host:
4
        return None
5
    url = "http://" + xen_host['host.name']
6
    log.info("\nXen Server host: " + xen_host['host.name'] + "\n")
7
    try:
8
        session = XenAPI.Session(url)
9
        session.xenapi.login_with_password(xen_host['host.user'], xen_host['host.password'])
10
    except XenAPI.Failure as f:
11
        error = "Failed to acquire a session: {}".format(f.details)
12
        log.error(error)
13
        return error
14
    return session


 

List VMs

Java
 




xxxxxxxxxx
1
36


 
1
def list_vms(session):
2
    vm_count = 0
3
    vms = session.xenapi.VM.get_all()
4
    log.info("Server has {} VM objects (this includes templates):".format(len(vms)))
5
    log.info("-----------------------------------------------------------")
6
    log.info("S.No.,VMname,PowerState,Vcpus,MaxMemory,Networkinfo,Description")
7
    log.info("-----------------------------------------------------------")
8
 
          
9
    vm_details = []
10
 
          
11
    for vm in vms:
12
        network_info = 'N/A'
13
        record = session.xenapi.VM.get_record(vm)
14
        if not (record["is_a_template"]) and not (record["is_control_domain"]):
15
            log.debug(record)
16
            vm_count = vm_count + 1
17
            name = record["name_label"]
18
            name_description = record["name_description"]
19
            power_state = record["power_state"]
20
            vcpus = record["VCPUs_max"]
21
            memory_static_max = record["memory_static_max"]
22
            if record["power_state"] != 'Halted':
23
                ip_ref = session.xenapi.VM_guest_metrics.get_record(record['guest_metrics'])
24
                network_info = ','.join([str(elem) for elem in ip_ref['networks'].values()])
25
            else:
26
                continue  # Listing only Running VMs
27
 
          
28
            vm_info = {'name': name, 'power_state': power_state, 'vcpus': vcpus,
29
                       'memory_static_max': memory_static_max, 'networkinfo': network_info,
30
                       'name_description': name_description}
31
            vm_details.append(vm_info)
32
            log.info(vm_info)
33
 
          
34
    log.info("Server has {} VM objects and {} templates.".format(vm_count, len(vms) - vm_count))
35
    log.debug(vm_details)
36
    return vm_details


 

Create VM

Java
 




xxxxxxxxxx
1
143


 
1
def create_vm(session, os_name, template, new_vm_name, cpus="default", maxmemory="default",
2
              expiry_minutes=MAX_EXPIRY_MINUTES):
3
    error = ''
4
    vm_os_name = ''
5
    vm_ip_addr = ''
6
    prov_start_time = time.time()
7
    try:
8
        log.info("\n--- Creating VM: " + new_vm_name + " using " + template)
9
        pifs = session.xenapi.PIF.get_all_records()
10
        lowest = None
11
        for pifRef in pifs.keys():
12
            if (lowest is None) or (pifs[pifRef]['device'] < pifs[lowest]['device']):
13
                lowest = pifRef
14
        log.debug("Choosing PIF with device: {}".format(pifs[lowest]['device']))
15
        ref = lowest
16
        mac = pifs[ref]['MAC']
17
        device = pifs[ref]['device']
18
        mode = pifs[ref]['ip_configuration_mode']
19
        ip_addr = pifs[ref]['IP']
20
        net_mask = pifs[ref]['IP']
21
        gateway = pifs[ref]['gateway']
22
        dns_server = pifs[ref]['DNS']
23
        log.debug("{},{},{},{},{},{},{}".format(mac, device, mode, ip_addr, net_mask, gateway,
24
                                                dns_server))
25
        # List all the VM objects
26
        vms = session.xenapi.VM.get_all_records()
27
        log.debug("Server has {} VM objects (this includes templates)".format(len(vms)))
28
 
          
29
        templates = []
30
        all_templates = []
31
        for vm in vms:
32
            record = vms[vm]
33
            res_type = "VM"
34
            if record["is_a_template"]:
35
                res_type = "Template"
36
                all_templates.append(vm)
37
                # Look for a given template
38
                if record["name_label"].startswith(template):
39
                    templates.append(vm)
40
                    log.debug(" Found %8s with name_label = %s" % (res_type, record["name_label"]))
41
 
          
42
        log.debug("Server has {} Templates and {} VM objects.".format(len(all_templates),
43
                                                                      len(vms) - len(
44
                                                                          all_templates)))
45
 
          
46
        log.debug("Choosing a {} template to clone".format(template))
47
        if not templates:
48
            log.error("Could not find any {} templates. Exiting.".format(template))
49
            sys.exit(1)
50
 
          
51
        template_ref = templates[0]
52
        log.debug("  Selected template: {}".format(session.xenapi.VM.get_name_label(template_ref)))
53
 
          
54
        # Retries when 169.x address received
55
        ipaddr_max_retries = 3
56
        retry_count = 1
57
        is_local_ip = True
58
        vm_ip_addr = ""
59
        while is_local_ip and retry_count != ipaddr_max_retries:
60
            log.info("Installing new VM from the template - attempt #{}".format(retry_count))
61
            vm = session.xenapi.VM.clone(template_ref, new_vm_name)
62
 
          
63
            network = session.xenapi.PIF.get_network(lowest)
64
            log.debug("Chosen PIF is connected to network: {}".format(
65
                session.xenapi.network.get_name_label(network)))
66
            vifs = session.xenapi.VIF.get_all()
67
            log.debug(("Number of VIFs=" + str(len(vifs))))
68
            for i in range(len(vifs)):
69
                vmref = session.xenapi.VIF.get_VM(vifs[i])
70
                a_vm_name = session.xenapi.VM.get_name_label(vmref)
71
                log.debug(str(i) + "." + session.xenapi.network.get_name_label(
72
                    session.xenapi.VIF.get_network(vifs[i])) + " " + a_vm_name)
73
                if a_vm_name == new_vm_name:
74
                    session.xenapi.VIF.move(vifs[i], network)
75
 
          
76
            log.debug("Adding non-interactive to the kernel commandline")
77
            session.xenapi.VM.set_PV_args(vm, "non-interactive")
78
            log.debug("Choosing an SR to instantiate the VM's disks")
79
            pool = session.xenapi.pool.get_all()[0]
80
            default_sr = session.xenapi.pool.get_default_SR(pool)
81
            default_sr = session.xenapi.SR.get_record(default_sr)
82
            log.debug("Choosing SR: {} (uuid {})".format(default_sr['name_label'], default_sr['uuid']))
83
            log.debug("Asking server to provision storage from the template specification")
84
            description = new_vm_name + " from " + template + " on " + str(datetime.datetime.utcnow())
85
            session.xenapi.VM.set_name_description(vm, description)
86
            if cpus != "default":
87
                log.info("Setting cpus to " + cpus)
88
                session.xenapi.VM.set_VCPUs_max(vm, int(cpus))
89
                session.xenapi.VM.set_VCPUs_at_startup(vm, int(cpus))
90
            if maxmemory != "default":
91
                log.info("Setting memory to " + maxmemory)
92
                session.xenapi.VM.set_memory(vm, maxmemory)  # 8GB="8589934592" or 4GB="4294967296"
93
            session.xenapi.VM.provision(vm)
94
            log.info("Starting VM")
95
            session.xenapi.VM.start(vm, False, True)
96
            log.debug("  VM is booting")
97
 
          
98
            log.debug("Waiting for the installation to complete")
99
 
          
100
            # Get the OS Name and IPs
101
            log.info("Getting the OS Name and IP...")
102
            config = read_config()
103
            vm_network_timeout_secs = int(config.get("common", "vm.network.timeout.secs"))
104
            if vm_network_timeout_secs > 0:
105
                TIMEOUT_SECS = vm_network_timeout_secs
106
 
          
107
            log.info("Max wait time in secs for VM OS address is {0}".format(str(TIMEOUT_SECS)))
108
            if "win" not in template:
109
                maxtime = time.time() + TIMEOUT_SECS
110
                while read_os_name(session, vm) is None and time.time() < maxtime:
111
                    time.sleep(1)
112
                vm_os_name = read_os_name(session, vm)
113
                log.info("VM OS name: {}".format(vm_os_name))
114
            else:
115
                # TBD: Wait for network to refresh on Windows VM
116
                time.sleep(60)
117
 
          
118
            log.info("Max wait time in secs for IP address is " + str(TIMEOUT_SECS))
119
            maxtime = time.time() + TIMEOUT_SECS
120
            # Wait until IP is not None or 169.xx (when no IPs available, this is default) and timeout
121
            # is not reached.
122
            while (read_ip_address(session, vm) is None or read_ip_address(session, vm).startswith(
123
                    '169')) and \
124
                    time.time() < maxtime:
125
                time.sleep(1)
126
            vm_ip_addr = read_ip_address(session, vm)
127
            log.info("VM IP: {}".format(vm_ip_addr))
128
 
          
129
            if vm_ip_addr.startswith('169'):
130
                log.info("No Network IP available. Deleting this VM ... ")
131
                record = session.xenapi.VM.get_record(vm)
132
                power_state = record["power_state"]
133
                if power_state != 'Halted':
134
                    session.xenapi.VM.hard_shutdown(vm)
135
                delete_all_disks(session, vm)
136
                session.xenapi.VM.destroy(vm)
137
                time.sleep(5)
138
                is_local_ip = True
139
                retry_count += 1
140
            else:
141
                is_local_ip = False
142
 
          
143
        log.info("Final VM IP: {}".format(vm_ip_addr))



Delete VM

Java
 




xxxxxxxxxx
1
34


 
1
def delete_vm(session, vm_name):
2
    log.info("Deleting VM: " + vm_name)
3
    delete_start_time = time.time()
4
    vm = session.xenapi.VM.get_by_name_label(vm_name)
5
    log.info("Number of VMs found with name - " + vm_name + " : " + str(len(vm)))
6
    for j in range(len(vm)):
7
        record = session.xenapi.VM.get_record(vm[j])
8
        power_state = record["power_state"]
9
        if power_state != 'Halted':
10
            # session.xenapi.VM.shutdown(vm[j])
11
            session.xenapi.VM.hard_shutdown(vm[j])
12
 
          
13
        # print_all_disks(session, vm[j])
14
        delete_all_disks(session, vm[j])
15
 
          
16
        session.xenapi.VM.destroy(vm[j])
17
 
          
18
        delete_end_time = time.time()
19
        delete_duration = round(delete_end_time - delete_start_time)
20
 
          
21
        # delete from CB
22
        uuid = record["uuid"]
23
        doc_key = uuid
24
        cbdoc = CBDoc()
25
        doc_result = cbdoc.get_doc(doc_key)
26
        if doc_result:
27
            doc_value = doc_result.value
28
            doc_value["state"] = 'deleted'
29
            current_time = time.time()
30
            doc_value["deleted_time"] = current_time
31
            if doc_value["created_time"]:
32
                doc_value["live_duration_secs"] = round(current_time - doc_value["created_time"])
33
            doc_value["delete_duration_secs"] = delete_duration
34
            cbdoc.save_dynvm_doc(doc_key, doc_value)


 

Historic Usage of VMs

It is better to maintain the history of all the VMs created and terminated along with other useful data. Here is the example of JSON document stored in the Couchbase, a free NoSQL database server. Insert a new document using the key as the xen opac reference uuid whenever a new VM is provisioned and update the same whenever VM is terminated. Track the live usage time of the VM and also how the provisioning/termination done by each user.

Java
 




xxxxxxxxxx
1
17


 
1
       record = session.xenapi.VM.get_record(vm)
2
       uuid = record["uuid"]
3
 
          
4
       # Save as doc in CB
5
        state = "available"
6
        username = new_vm_name
7
        pool = "dynamicpool"
8
        doc_value = {"ipaddr": vm_ip_addr, "origin": xen_host_description, "os": os_name,
9
                     "state": state, "poolId": pool, "prevUser": "", "username": username,
10
                     "ver": "12", "memory": memory_static_max, "os_version": vm_os_name,
11
                     "name": new_vm_name, "created_time": prov_end_time,
12
                     "create_duration_secs": create_duration, "cpu": vcpus, "disk": disks_info}
13
        # doc_value["mac_address"] = mac_address
14
        doc_key = uuid
15
 
          
16
        cb_doc = CBDoc()
17
        cb_doc.save_dynvm_doc(doc_key, doc_value)


 

Java
 




xxxxxxxxxx
1
31


1
class CBDoc:
2
    def __init__(self):
3
        config = read_config()
4
        self.cb_server = config.get("couchbase", "couchbase.server")
5
        self.cb_bucket = config.get("couchbase", "couchbase.bucket")
6
        self.cb_username = config.get("couchbase", "couchbase.username")
7
        self.cb_userpassword = config.get("couchbase", "couchbase.userpassword")
8
        try:
9
            self.cb_cluster = Cluster('couchbase://' + self.cb_server)
10
            self.cb_auth = PasswordAuthenticator(self.cb_username, self.cb_userpassword)
11
            self.cb_cluster.authenticate(self.cb_auth)
12
            self.cb = self.cb_cluster.open_bucket(self.cb_bucket)
13
        except Exception as e:
14
            log.error('Connection Failed: %s ' % self.cb_server)
15
            log.error(e)
16
 
          
17
    def get_doc(self, doc_key):
18
        try:
19
            return self.cb.get(doc_key)
20
        except Exception as e:
21
            log.error('Error while getting doc %s !' % doc_key)
22
            log.error(e)
23
 
          
24
    def save_dynvm_doc(self, doc_key, doc_value):
25
        try:
26
            log.info(doc_value)
27
            self.cb.upsert(doc_key, doc_value)
28
            log.info("%s added/updated successfully" % doc_key)
29
        except Exception as e:
30
            log.error('Document with key: %s saving error' % doc_key)
31
            log.error(e)



Java
 




xxxxxxxxxx
1
20


 
1
"dynserver-pool": {
2
  "cpu": "6",
3
  "create_duration_secs": 65,
4
  "created_time": 1583518463.8943903,
5
  "delete_duration_secs": 5,
6
  "deleted_time": 1583520211.8498628,
7
  "disk": "75161927680",
8
  "ipaddr": "x.x.x.x",
9
  "live_duration_secs": 1748,
10
  "memory": "6442450944",
11
  "name": "Win2019-Server-1node-DynVM",
12
  "origin": "s827",
13
  "os": "win16",
14
  "os_version": "",
15
  "poolId": "dynamicpool",
16
  "prevUser": "",
17
  "state": "deleted",
18
  "username": "Win2019-Server-1node-DynVM",
19
  "ver": "12"
20
}


 

Configuration

The Dynamic VM server manager service configuration such as Couchbase server, xenhost servers, template details, default expiry, and network timeout values can be maintained in a simple .ini format. Any new Xen Host received, then just add as a separate section. The config is dynamically loaded without restarting the Dynamic VM SM service.

Sample config file: .dynvmservice.ini

Java
 




xxxxxxxxxx
1
45


 
1
[couchbase]
2
couchbase.server=<couchbase-hostIp>
3
couchbase.bucket=<bucket-name>
4
couchbase.username=<username>
5
couchbase.userpassword=<password>
6
 
          
7
[common]
8
vm.expiry.minutes=720
9
vm.network.timeout.secs=400
10
 
          
11
[xenhost1]
12
host.name=<xenhostip1>
13
host.user=root
14
host.password=xxxx
15
host.storage.name=Local Storage 01
16
centos.template=tmpl-cnt7.7
17
centos7.template=tmpl-cnt7.7
18
windows.template=tmpl-win16dc - PATCHED (1)
19
centos8.template=tmpl-cnt8.0
20
oel8.template=tmpl-oel8
21
deb10.template=tmpl-deb10-too
22
debian10.template=tmpl-deb10-too
23
ubuntu18.template=tmpl-ubu18-4cgb
24
suse15.template=tmpl-suse15
25
 
          
26
[xenhost2]
27
host.name=<xenhostip2>
28
host.user=root
29
host.password=xxxx
30
host.storage.name=ssd
31
centos.template=tmpl-cnt7.7
32
windows.template=tmpl-win16dc - PATCHED (1)
33
centos8.template=tmpl-cnt8.0
34
oel8.template=tmpl-oel8
35
deb10.template=tmpl-deb10-too
36
debian10.template=tmpl-deb10-too
37
ubuntu18.template=tmpl-ubu18-4cgb
38
suse15.template=tmpl-suse15
39
 
          
40
[xenhost3]
41
...
42
 
          
43
 
          
44
[xenhost4]
45
...


 

Examples

Sample REST API Calls Using curl

Java
 




xxxxxxxxxx
1
54


 
1
$ curl 'http://127.0.0.1:5000/showall'
2
{
3
  "1": [
4
    {
5
      "memory_static_max": "6442450944",
6
      "name": "tunable-rebalance-out-Apr-29-13:43:59-7.0.0-19021",
7
      "name_description": "tunable-rebalance-out-Apr-29-13:43:59-7.0.0-19021 from tmpl-win16dc - PATCHED (1) on 2020-04-29 20:43:33.785778",
8
      "networkinfo": "172.23.137.20,172.23.137.20,fe80:0000:0000:0000:0585:c6e8:52f9:91d1",
9
      "power_state": "Running",
10
      "vcpus": "6"
11
    },
12
    {
13
      "memory_static_max": "6442450944",
14
      "name": "nserv-nserv-rebalanceinout_P0_Set1_compression-Apr-29-06:36:12-7.0.0-19022",
15
      "name_description": "nserv-nserv-rebalanceinout_P0_Set1_compression-Apr-29-06:36:12-7.0.0-19022 from tmpl-win16dc - PATCHED (1) on 2020-04-29 13:36:53.717776",
16
      "networkinfo": "172.23.136.142,172.23.136.142,fe80:0000:0000:0000:744d:fd63:1a88:2fa8",
17
      "power_state": "Running",
18
      "vcpus": "6"
19
    }
20
  ],
21
  "2": [
22
    ..
23
  ]
24
  "3": [
25
    ..
26
  ]
27
  "4": [
28
    ..
29
  ]
30
  "5": [
31
    ..
32
  ]
33
  "6": [
34
    ..
35
  ]
36
}
37
 
          
38
$ curl 'http://127.0.0.1:5000/getavailablecount/windows'
39
2
40
$ curl 'http://127.0.0.1:5000/getavailablecount/centos'
41
10 
42
$ curl 'http://127.0.0.1:5000/getservers/demoserver?os=centos&count=3'
43
["172.23.137.73", "172.23.137.74", "172.23.137.75"]
44
$ curl 'http://127.0.0.1:5000/getavailablecount/centos'
45
7
46
$ curl 'http://127.0.0.1:5000/releaseservers/demoserver?os=centos&count=3'
47
[
48
  "demoserver1",
49
  "demoserver2",
50
  "demoserver3"
51
]
52
$ curl 'http://127.0.0.1:5000/getavailablecount/centos'
53
10 
54
$


 

Jenkins Jobs With Single VM

Java
 




xxxxxxxxxx
1


 
1
curl -s -o ${BUILD_TAG}_vm.json "http://<host:port>/getservers/${BUILD_TAG}?os=windows&format=detailed"
2
VM_IP_ADDRESS="`cat ${BUILD_TAG}_vm.json |egrep ${BUILD_TAG}|cut -f2 -d':'|xargs`"
3
if [[ $VM_IP_ADDRESS =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]]; then
4
  echo Valid IP received
5
else
6
  echo NOT a valid IP received
7
  exit 1
8
fi


 

Jenkins Jobs With Multiple VMs Needed

Java
 




xxxxxxxxxx
1
22


 
1
curl -s -o ${BUILD_TAG}.json "${SERVER_API_URL}/getservers/${BUILD_TAG}?os=${OS}&count=${COUNT}&format=detailed"
2
cat ${BUILD_TAG}.json
3
VM_IP_ADDRESS="`cat ${BUILD_TAG}.json |egrep ${BUILD_TAG}|cut -f2 -d':'|xargs|sed 's/,//g'`"
4
echo $VM_IP_ADDRESS
5
ADDR=()
6
INDEX=1
7
for IP in `echo $VM_IP_ADDRESS`
8
do
9
  if [[ $IP =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]]; then
10
    echo Valid IP=$IP received
11
    ADDR[$INDEX]=${IP}
12
    INDEX=`expr ${INDEX} + 1`
13
  else
14
    echo NOT a valid IP=$IP received
15
    exit 1
16
  fi
17
done
18
 
          
19
echo ${ADDR[1]}
20
echo ${ADDR[2]}
21
echo ${ADDR[3]}
22
echo ${ADDR[4]}


 

Key Considerations

Here are a few of my observations noted during the process and it is better to handle to make it more reliable.

  1. Handle Storage name/ID different among different Xen Hosts
    • Keep track of VM storage device name in the service input config file.
  2. Handle partial templates only available on some Xen Hosts while provisioning
  3. When Network IPs not available and Xen APIs gets the default 169.254.xx.yy on Windows. Wait until getting the non 169 address or timeout.
  4. Release servers should ignore os template as some of the templates might not be there Xen Hosts
  5. Provision on a specific given Xen Host reference
  6. Handle No IPs available or not getting network IPs for some of the VMs created.
    • Plan to have a different subnet for dynamic VMs targeted Xen Hosts. The default network DHCP IP lease expiry might be in days (say 7 days) and no new IPs are provided.
  7. Handle the capacity check to count the in progress as Reserved IPs and should show less count than full at the moment. Otherwise, both in-progress and incoming requests might have issues. One or two VMs (cpus/memory/disk sizes) can be in buffer while creating and checking if any parallel requests.

References

Some of the key references that help while creating the dynamic VM server manager service.

  1. https://www.couchbase.com/downloads
  2. https://wiki.xenproject.org/wiki/XAPI_Command_Line_Interface
  3. https://xapi-project.github.io/xen-api/
  4. https://docs.citrix.com/en-us/citrix-hypervisor/command-line-interface.html
  5. https://github.com/xapi-project/xen-api-sdk/tree/master/python/samples
  6. https://www.citrix.com/community/citrix-developer/citrix-hypervisor-developer/citrix-hypervisor-developing-products/citrix-hypervisor-staticip.html
  7. https://docs.ansible.com/ansible/latest/modules/xenserver_guest_module.html
  8. https://github.com/terra-farm/terraform-provider-xenserver
  9. https://github.com/xapi-project/xen-api/blob/master/scripts/examples/python/renameif.py
  10. https://xen-orchestra.com/forum/topic/191/single-device-not-reporting-ip-on-dashboard/14
  11. https://xen-orchestra.com/blog/xen-orchestra-from-the-cli/
  12. https://support.citrix.com/article/CTX235403

Hope you had a good reading time!

Disclaimer: Please view this as a reference if you are dealing with Xen Hosts. Feel free to share if you learned something new that can help us. Your positive feedback is appreciated!


Thanks to Raju Suravarjjala, Ritam Sharma, Wayne Siu, Tom Thrush, James Lee for their help during the process.

Topics:
best practices ,couchbase ,devops ,dynamic allocation ,flask ,infrastructure automation ,python 3 ,xen servers

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}