Infrastructure Automation with Open Source Tools
Infrastructure Automation with Open Source Tools
Infrastructure automation opens the door for developers to rapidly provision new environments, and this list of open source tools can help you get started.
Join the DZone community and get the full member experience.Join For Free
Enterprises across the globe are implementing infrastructure automation to save time, eliminate manual error/variability, and reduce costs. Automation opens the door for developers to rapidly provision new environments. Cloud architects, support executives and others looking at infrastructure automation across the enterprise
It is the administration and management of physical and virtual assets, data, and software to support all IT Operations in the facility. This structure is required to run all Business Operations smoothly. The main goal is to minimize downtime and maintain business productivity.
These are the main elements and services for Infrastructure Management:
- Asset lifecycle
- Capacity monitoring/planning
- Network utilization
The Need for Infrastructure Management
Earlier, setting up new infrastructure meant stacking physical servers, configuring network cables, and housing hardware in a capable data center. In the cloud, setting up more performance efficient, cost-effective, and secure infrastructure can all be done using software just by clicking buttons.
This is done by writing the services as IaC for Infrastructure. The use of infrastructure as code means IT infrastructure can be fully provisioned from source code rather than manually. IaC opens the door to the complete automation and configuration of infrastructure, with cloud elements such as servers, networks, and databases being treated similarly to the software.
This all can be done in our Infrastructure with Open Source Tools and services.
IaC is made more accessible today through a comprehensive range of tools available for automating the entire provisioning process.
Quick and secure deployment is the key to successful Infrastructure Management.
Tools Required for Infrastructure Management
4. Foreman/Smart Proxy
Advantages of Tools
- Discover, provision and upgrade your entire bare-metal infrastructure
- Create and manage instances in a virtualization environment and across private and public clouds
- Install operating systems via PXE, local media or from templates or images
- Control and gather reports from your configuration management software
- Group your hosts and manage them in bulk, regardless of location
- Review historical changes for auditing or troubleshooting
- Web user interface, JSON REST API and CLI for Linux
- "Push Button" Automation
- Remote Command Execution
- Ansible ® is an agentless software provisioning, configuration management, and application management tool that deploys modules to nodes over SSH.
- Terraform is designed to efficiently provision and de-provision public cloud infrastructure with extensive support for AWS. Many DevOps and support teams will use Terraform to spin up a cloud on day 0 and then use other tools to automate cloud image security and the ongoing day two maintenance of the infrastructure.
- AWX is an enterprise framework for controlling, securing, and managing your Ansible automation with a UI and RESTful API. Role-based access control keeps environments secure, and teams efficient. Non-privileged users can safely deploy entire applications with push-button deployment access.
- Foreman helps system administrators manage servers throughout their lifecycle, from provisioning and configuration to orchestration and monitoring. With Foreman, you can quickly deploy applications, and proactively manage change, both on-premise with VMs and bare-metal or in the cloud. Foreman provides comprehensive, interaction facilities including a web frontend, CLI, and RESTful API which enables you to build higher-level business logic on top of a solid foundation.
- Smart Proxy as the name suggests it's just proxy for Foreman and can be located near to the hosts. If there are multiple server locations. Smart Proxy can be placed in each location to improve latencies and proper segregation in DMZ and PCI environment.
- OpenLDAP open-source implementation of the Lightweight Directory Access Protocol for authenticating the user in the AWX and Foreman environment.
- GIT is a distributed version control system for tracking changes in source code during software development
Getting Started with Automation
Use SSH keys wherever it's possible to connect to other nodes. Make sure to implement proper monitoring and alert system for managing your environment.
Need to write the code in Ansible YAML language. It's a very simple programming language. Playbooks are Ansible's configuration, deployment, and orchestration language. It contains configurations for remote machines. These remote machines are saved in inventory files.
Save all the playbook in the Version Control System, Only authorized user will have access to these playbooks.
Foreman will be set up and connects to all the hosts with the help of smart proxy in every data center to monitor the hosts. Content views will be created for every package which has to be pushed to hosts.
AWX will pull the inventory information from Foreman which gives the real-time data for hosts. Now AWX will connect to GIT and pull the playbook to be run on specific hosts which we got it from the Foreman server. On click of the AWX template, it pushes the command to the host to run patch or upgrades to specific servers. Content can be utilized from the Foreman repository which is published after creating the content view.
The log can be added in LogInsight tool
Step 5:Foreman can be utilized for Discover, provision, and upgrade your entire bare-metal infrastructure. Install operating systems via PXE, local media or from templates or images
AWX can be utilized as 3 cluster nodes to improve performance. It contains the AWX-WEB module to handle all the UI redirections. AWX-Task and RabbitMQ module manages the scheduling of all messages and tasks. Memcache caches client information.
In our scenario, we will have an External PostgreSQL database it stores all metadata and transaction information. In the case of the failover scenario, another PostgreSQL server will be required with replicated data from the Master.
In secure environments like DMZ and PCI will have a lower version of AWX which is called Isolated Node. The isolated node will not have any UI and it will execute the task from the main AWX node.
Foreman will monitor all the hosts with the help of Smar Proxies deployed in every data center
Detailed Architecture for Foreman.
- Katello is a Foreman plug-in for subscription and repository management. It provides a means to subscribe to Red Hat repositories and download content.
- Candlepin is a service within Katello that handles subscription management.
- Pulp is a service within Katello that handles repository and content management.
- Celery is an open-source asynchronous task queue or job queue which is based on distributed message passing. While it supports scheduling, its focus is on operations in real-time
- Pulp_Worker: This component is responsible for performing asynchronous tasks, such as sync and publish.
- Pulp_Celerybeat: This process is responsible for queuing scheduled tasks and is responsible for monitoring the availability of worker
- Pulp_Resource_Manager: This is a singleton worker that is responsible for assigning tasks to workers based on which resource they need to reserve.
- Pulp_Streamer: Determine where the requested content can be found
- Squid: Caches/Stream the content
- Pulp Streamer returns the downloaded bits to Squid as it receives them. This ensures the client does not time out the connection since Squid will in turn stream the content
- Qpidd: Receive, Store and send message
Foreman Task Execution Flow
Whenever there is a request for a task in Foreman it stores in Celery asynchronous task queue. Pulp Celery Beat component schedule & queues this task. At the same time, it monitors the available task worker.
Pulp Resource Manager is a singleton component that assigns the task to the pulp workers based on the resources they need to reserve.
Pulp Streamer returns the downloaded bits to Squid as it receives them. This ensures the client does not time out the connection since Squid will, in turn, caches the content
Every enterprise needs infrastructure automation without any human intervention. As the demand for faster delivery is increasing day by day enterprises are looking for foolproof solutions for Infrastructure Automation. Infrastructure automation streamlines IT operations, improves productivity, increases the pace of introducing new technologies and applications, and lowers costs associated with downtime and compliance/regulatory risks. Ansible AWX is a powerful tool for automating the configuration of computers and orchestrating the entire application across the network, no matter where it is deployed. Ansible AWX has web-based UI which helps in orchestration, scalability, reliability, and consistency with your Infrastructure environment. Task automation is done by running Ansible Playbooks in AWX. If your infrastructure lies in the cloud environment Terraform can help to provision cloud infrastructure. Foreman is a life cycle environment and you can provision bare-metal machine on-prem or on the cloud. We can get the run time status/facts info for all the nodes in your enterprise. If enterprises do not want to spend money on licensed products and they have the support team it's better to utilize open source tools for their Infrastructure Automation.
Opinions expressed by DZone contributors are their own.