Let's continue our discussion of the essential skills to look for when hiring a DevOps engineer. If you haven't read Part I yet, you can check it out here.
System Tools and Methods
To get an automation project up and running, a DevOps engineer builds new things such as configuration objects in an application and code snippets of full-blown programs. However, a major part of the work is gluing many things together at the system level on the given infrastructure. Such efforts are not different from traditional system integration work and, in my opinion, the ingenuity of an engineer at this level determines his or her real value on the team. It is easy to find cookbooks, recipes, and best practices for vendor-supported tools, but it would take experience working on diverse projects to gain the necessary skill set to implement robust integrations that have to work reliably in production.
Important system-level tools and techniques are listed here. The engineer should have knowledge about the following.
- Users and groups on Linux.
- Use of service accounts for automation.
- Sudo commands, /etc/sudoers files, and passwordless access.
- Using LDAP and AD for access management.
- Remote access using SSH.
- SSH keys and related topics.
- SCP, SFTP, and related tools.
- SSH key formats.
- Managing access using configuration management tools.
- Use of GPG for password encryption.
- Tools for password management such as KeePass.
- MD5 for encryption.
- Remote access with authentication from automation scripts.
- Managing API keys.
- Jenkins plugins for password management.
- Basics of compilers such as GCC and Javac.
- Make and Makefile, Ant, Maven, Gradle, etc.
- Code libraries in C++, Java, PHP, Perl, Ruby, Python, etc.
- Build artifacts such as JAR and WAR files.
- Running builds from Jenkins.
- Packaging files: ZIP, TAR, GZIP, etc.
- Packaging for deployment: RPM, Debian, DNF, Zypper, etc.
- Packaging for the cloud: AWS AMI, VMWare template, etc.
- Use of Packer.
- Docker and containers for microservices.
- Use of artifacts repository: Distribution and release of builds; meeting build and deployment dependencies.
- Serving artifacts from shared storage volume.
- Mounting locations from cloud storage services such AWS S3.
- Artifactory as artifacts server.
- SCP, Rsync, FTP, and their SSL counterparts.
- Via shared storage.
- File transfer with cloud storage services such as AWS S3.
- Code pushing using system level file transfer tools.
- Scripting using SSH libraries such as Paramiko.
- Orchestrating code pushes using configuration management tools.
- Use of crontab.
- Running jobs in the background; use of Nohup.
- Use of screen to launch long-running jobs.
- Jenkins as a process manager.
Files and Storage
- Typical uses of find, DF, DU, etc.
- A comparison of popular distributions.
- Checking OS release and system info.
- Package management differences.
- Typical uses of SED, AWK, GREP, TR, etc.
- Scripting using Perl.
- Regular expressions.
- Support for regular expressions in Perl and Python.
Sample usages and steps to install these tools:
Programming Primer for DevOps
One of the attributes that helps differentiate a DevOps engineer from other members in the operations team, like sysadmins, DBAs, and operations support staff, is his or her ability to write code. The coding and scripting skill is just one of the tools in the DevOps toolbox, but it's a powerful one that a DevOps engineer would maintain as part of practicing his or her trade.
Coding is the last resort when things cannot be integrated by configuring and tweaking the applications and tools that are used in an automation project.
Bash Scripting Essentials
Many times, a few lines of bash script could be the best glue code integrating two components in the whole software system. DevOps engineer should have basic shell scripting skills and Bash is the most popular right now.
If a script has to deal with external systems and components or it's more than just a few lines of command-lines and dealing with fairly complex logic, it might be better to write that script in an advanced scripting language like Python, Perl, or Ruby.
Knowledge of Python would make your life easier when dealing with DevOps applications such as Ansible, which uses Python syntax to define data structures and implement conditionals for defining configurations.
One of the categories of projects a DevOps engineer would end up doing is building dashboards. Though dashboarding features are found with most of the DevOps tools, those are specific to the application, and there will be a time when you may require to have a general purpose dashboard with more dynamic content than just static links and text.
Another requirement is to build web UI for provisioning tools to present those as self-service tools to user groups.
Almost every application and tool that is used for building, deploying, and maintaining software systems uses configuration files. While manual reading of these files might not require any expertise, a DevOps engineer should know how config files in such formats are created and parsed programmatically.
A DevOps engineer should have a good understanding of these formats:
The engineer should also know how these formats are parsed in his/her favorite scripting language.
The wide acceptance of REST API as a standard to expose features that other applications can use for system integration made it a feature requirement for any application that wants to be taken seriously. The knowledge of using REST API has become an important skill for DevOps engineer.
- HTTP/HTTPS: REST APIs are based on HTTP/HTTPS protocol and a solid understanding of its working is required. Knowledge of HTTP headers, status codes, and main verbs GET, POST, and PUT.
- REST API basics: Normal layout of APIs defined for an application.
- Curl and Wget: Command line tools to access REST API and HTTP URLs. Some knowledge of the support available for HTTP protocol in scripting languages will be useful and that would be an indication of working with REST APIs.
- Authentication methods: Cookie-based and OAuth authentication; API keys; use of If-Match and If-None-Match set of HTTP headers for updates.
- API management tools: If the application you support provides an API for the users, most probably, its usage will be managed by some API Gateway tool. Though not an essential skill, experience in this area would be good if one works on the API provider side.
Programming With Data Repositories
There was a time when mere knowledge of programming with RDBMS was enough for an application developer and system integrator to manage application data. Now with the wide adoption of Big Data platform like Hadoop and NOSQL systems to process and store data, a DevOps engineer needs varied requirements, from one project to another. Core skills are the following:
- RDBMS: MySQL, Oracle, SQL Server, Postgres, etc. are still used and knowledge of one or more is important.
- Setting up and configuring MySQL: As an open-source database used with many other tools in the DevOps toolchain, I consider this a basic requirement for a DevOps engineer. If one hasn’t done this, he or she might not have done enough yet.
- ANSI SQL: Basic DML and aggregation functions.
- ODBC: Tools depend on this to access a variety of data repositories and therefore the configuration of ODBC on Linux is needed sometimes.
- Running queries from a Bash script: How to run a database query via a database client from a Bash script and use the output. MySQL is a good example.
- Database access from Perl/PHP/Python: All the major scripting languages provide modules to access databases and that can be used to write robust automation scripts. Examples are Perl DBI and Python’s MySQLdb module.
- NOSQL: There is a plethora of NOSQL systems, but knowledge of popular applications among those, Cassandra and MongoDB, would be better. The skills should be similar to those discussed above for RDBMS.
- Hadoop: The default platform for Big Data processing and storage. Basic knowledge of Hadoop suite of tools (HDFS, HBase, Hive, Pig, etc.) would be useful.
Programming for Cloud
Those who have built cloud infrastructure with a focus on automation and versioning should know some of these (or similar) tools:
- cloud-init: Cloud-init can be used to configure a virtual machine when it is spun up. This is very useful when a node is spun up from a machine image with baseline or even application software already baked in.
- AWS CLI: If the application runs on Amazon AWS, knowledge of AWS CLI is needed, which would be handy to put together simple automation scripts.
- Terraform: HashiCorp’s Terraform is an important tool if the focus would be to provision infrastructure as code (IaaS). Using this, infrastructure can be configured independently of the target cloud or virtualization platform.
- Packer: It can be used to build machine images for a variety of virtualization technologies and cloud platforms like AWS and it is useful if the infrastructure is provisioned in a mixed or hybrid cloud environment.
In a rush to get things rolled out, one of the things left half-done is adding enough error handling in scripts. Automation scripts that are not robust can cause major production issues, which could impact the credibility of DevOps efforts itself. A DevOps engineer should be aware of the following best practices in error handling and logging:
- The importance of error handling in automated scripts.
- Error handling in Bash.
- Error handling in Python.
- Logging errors in application and system logs.
Looking at these requirements for a DevOps engineer, it is pretty clear that he or she should have a variety of skills to succeed at work. That brings up a hiring problem in finding skilled engineers. In theory, a template like this should be used only for assessing the current experience of a prospective hire. The needed skills can be picked up on the jobs that demand deep knowledge in certain areas. Therefore, the focus should be to hire smart engineers who have a track record of picking up new skills, rolling out innovative projects at work, and contributing to reputed open-source projects.