Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

The Magic of DevOps

DZone's Guide to

The Magic of DevOps

This article walks DevOps newcomers through basic processes to provide the foundation for powerful DevOps benefits you can apply in production.

· DevOps Zone ·
Free Resource

Learn more about how CareerBuilder was able to resolve customer issues 5x faster by using Scalyr, the fastest log management tool on the market. 

Where to start, what to do, how does it work...what is it? 

You may ask your self these questions when you hear and see a lot of articles, books, webinars, and seminars about microservices, Docker, Jenkins, Ansible, Kubernetes, OpenShift, Cloud Foundry, and DevOps. 

Wow, there are a lot of things to learn. I won’t start talking about the theory behind microservices, Docker, or DevOps. I am sure that you know what is it, but the question is, how does it work together?

In this article, I will be going with you step-by-step to show you how to start and apply some practices that can help you in getting started with the concepts of microservices and DevOps.

Disclaimer: All the actions that would be applied in this article are not the best practices in a real production environment; the goal is simply to show some basic techniques that could be powerful for development and testing environment. If you'd like to read about the best architecture for DevOps, unfortunately, this is not the best article for you.

Prerequisites

This article can be applied on a Unix or Linux machine; I’m using Ubuntu. Before you start, you need to install the following:

  1. Docker and docker-compose
  2. Create an account on hub.docker.com

After successful installation of Docker and creating an account on hub.docker.com, let me explain in simple words what will be our story:

I want to have some magic that starts after I push my code to the GitRepo (e.x. GitHub, GitLab, or whatever) and starts compiling my code, builds it, builds a Docker image, pushes it to hub.docker.com, and runs it on some machine. 

Image title

Let's Get Started

  1. We need to have a GitRepo to save and manage our work; we will use GitLab as a Docker container, so go to your terminal and type the following:
mkdir -p /home/dev/srv/gitlab/config
mkdir -p /home/dev/srv/gitlab/logs
mkdir -p /home/dev/srv/gitlab/data


docker run -d \
--hostname 192.168.100.9 \
    --publish 443:443 --publish 80:80 --publish 222:22 \
    --name gitlab \
    --restart always \
    --volume /home/dev/srv/gitlab/config:/etc/gitlab \
    --volume /home/dev/srv/gitlab/logs:/var/log/gitlab \
    --volume /home/dev/srv/gitlab/data:/var/opt/gitlab \
    gitlab/gitlab-ce

Change 192.168.100.9 to your IP address. Wait for 5 minutes until the installation of GitLab is completed, visit your http://yourIpAddress/, then the GitLab server will ask you to enter the root password - enter the root password, and after that you will be redirected to the login page. Enter username: root and password: whatYouHaveEnteredBefore. Go to users, create a new user, and enter the username and email, then save, go back to the users list, click edit user, and enter the newly created user password. Log out of the root account, and enter with the created user email and password.

Now let's add some code.

# go to your workspace dir, then
git clone https://github.com/yasser-kfupm/devops.git

#then
cd devops

sudo rm -r .git


Now, we will go to your GitLab and create a new project and add our files to it.

Image title

Then, a new page will be displayed with options how to start with your project; go back to your terminal and do the normal steps to create a project from existing code:

# your should be in devops dir, 
git init
git remote add origin http://192.168.100.9/deep01code/devops.git
git add .
git commit -m "Initial commit"
git push -u origin master


cd existing_folder
git init
git remote add origin http://192.168.100.9/deep01code/devops.git
git add .
git commit -m "Initial commit"
git push -u origin master


In Jenkins, there are many ways and countless plugins designed to handle different needs to accomplish CI/CD. If you try to search Jenkins, you will find many resources talk about it, however, I am interested in utilizing Jenkins in the simplest way that will help me in my local environment to build microservices easily. Let us complete what we started. I will use a dockerized container for Jenkins, and before we do that, we need to do some smaller tricks.

# to make sure that our containers will be able to utilize some files on our host
# machine I will change some privilages

chmod -R 777 /var/run/docker.sock
chmod -R 777 /var/lib/docker


I assume that you understand what my trick is here. I am planning to do a volume mount for Docker files on my host machine to be used by the Jenkins container. In the next step, we will build a special Docker image that will install sshpass and Docker. Go to your terminal and change your directory to any convenient place, then run: 

sudo nano Dockerfile

This will open an empty file. Copy and paste the following code:

FROM jenkins/jenkins:lts
USER root
RUN apt-get update && \
apt-get -y install apt-transport-https \
     ca-certificates \
     curl \
     sshpass \
     gnupg2 \
     software-properties-common && \
curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg > /tmp/dkey; apt-key add /tmp/dkey && \
add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
   $(lsb_release -cs) \
   stable" && \
apt-get update && \
apt-get -y install docker-ce
USER jenkins

Save the nano file.

docker build -t deep01code/xjenkins .

This will create a Docker image named deep01code/xjenkins. Now, let's run that image:

docker run --name xjenkins -p 8080:8080 -p 50000:50000  --restart always -d -v /var/lib/docker:/var/lib/docker  -v /var/run/docker.sock:/var/run/docker.sock -v /home/dev/docker-path:/var/jenkins_home deep01code/xjenkins

If you typed docker ps, you should see something similar to this:

docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                  PORTS                                                           NAMES
52c8b399b7e3        deep01code/xjenkins   "/sbin/tini -- /usr/…"   34 hours ago        Up 24 hours             0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp                xjenkins
ba7cc62c1271        dd7c0664f2e1          "/assets/wrapper"        40 hours ago        Up 24 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:222->22/tcp   gitlab

So, what's next? Now, go to Jenkins http://localhost:8080. Jenkins will ask you enter a secret password:

Image title

You can find the secret password by typing: 

docker logs xjenkins

#you should see it in the log, copy that password and paste

Jenkins will ask you to enter a username, password, and email. Fill in the information and continue. Another page will ask you to select the plugin- select the default option

Image title

After that, go to Jenkins plugins, search for the GitLab plugin, and install it.

Stop!

Hold your horses - let me tell you what is going on. The plan is simple: we have your code on GitLab. If a "git push" code is fired from our repo, we want to have some notification to Jenkins so it can start some magic. However, at the end of whatever Jenkins is doing, we want to deploy a microservice somewhere. OK...how did I do it?

The answer is simple. By remotely calling the server machine and calling "docker-compose up," the idea of docker-compose is to put your services in containers, and then inside docker-compose.yml, you specify these services. Each time you call "docker-compose up," Docker will pull the latest changes from dockerhub and apply them to your running containers. So, let us prepare our docker-copmose.yml file on our server.

At the end of your Jenkins pipline execution, I will push a Docker container to the hub named deep01code/devopsx.

#back to your terminal,

sudo mkdir -p /home/dev/deployment
sudo cd /home/dev/deployment
sudo nano docker-compose.yml

#copy and past the following code

version: "2"
services:
  web:
    image: deep01code/devopsx
    ports:
     - 9999:8080

Let us go back to Jenkins.

Jenkins somehow will know that there was some code pushed to GitLab (we will see how shortly) then Jenkins will start reading a file called a Jenkinsfile. In that file, we can define the stages and steps we would like to execute for our project (microservice or monolithic app). These stages can be customized as you require; for example, we can create a compile stage, build stage, test stage... and so on. The nice thing about Jenkins is that many plugins will help you to generate reports to show testing results and more than that.

So, the Jenkinsfile will be in the root directory of your project, and it will be like this:

pipeline {
  agent any

  triggers {
    pollSCM('* * * * *')
  }

  stages {

    stage("Compile") {
      steps {
      sh "./gradlew compileJava"
      }
    }


    stage("Build") {
      steps {
       sh "./gradlew build"
      }
    }

    stage("Docker build") {
      steps {
     sh "docker build -t deep01code/devopsx ."
      }
    }

    stage("Docker login") {
      steps {
    sh "docker login --username yourUserName --password yourPassword"
      }
    }

    stage("Docker push") {
      steps {
   sh "docker push deep01code/devopsx"
      }
    }

    stage("Deploy to staging") {
      steps {
        sh "sshpass -p 'dev' ssh  -o StrictHostKeyChecking=no  dev@192.168.100.9 'cd /home/dev/deploy/ && pwd && ls && /usr/bin/docker-compose up -d'"
      }
    }




  }
}

Notice that we are using the host machine Docker mapped to our Jenkins Docker container. If you try to go to the xjenkins container bash and run "docker ps," you will have the same output if you run the same command from the host machine.

Now, let us create our pipeline. Copy your GitLab project repo URL. Go to your Jenkins webpage http://localhost:8080, click on "new item," enter the item name, and select pipeline then "ok."

Image title

Caution: Make sure to enter an item name matching the name of the jar file that will be mentioned in the project Dockerfile, which is also in our project root directory.

FROM frolvlad/alpine-oraclejdk8:slim
ADD build/libs/xops-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT [ "java", "-jar", "app.jar" ]

After you click "OK," a new page will be displayed. Go to the Pipeline section and select the definition to be " Pipeline script from SCM," then enter the repo directory.

Image title

Go to the Build Triggers (here is how Jenkins knows if our code changes).

Image title

Then click "Apply" and "Save." By now, I think you are ready to see the magic happen. If all the steps were done correctly, I think you shall see your pipeline execution started, and it will be something like this:

Image title

If you try to check if anything is running other than your previous containers, type "docker ps."

You will see something new! Congratulations...you did it.

docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                  PORTS                                                           NAMES
ef89667592dc        deep01code/devopsx    "java -jar app.jar"      13 seconds ago      Up 12 seconds           0.0.0.0:9999->8080/tcp                                          deploy_web_1
52c8b399b7e3        deep01code/xjenkins   "/sbin/tini -- /usr/…"   36 hours ago        Up 27 hours             0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp                xjenkins
ba7cc62c1271        dd7c0664f2e1          "/assets/wrapper"        42 hours ago        Up 27 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:222->22/tcp   gitlab

As you can see, there is a Docker container running on port 9999 (this information was specified in the docker-compose.yml file). You can access it and test the REST endpoint on http://localhost:9999/exe, and it should return Hello DevOps.

I did my best to explain every step and make things clear for you. However, I am sure that you will face something wrong here or there, so be patient. Let me show you how many times I tried to run my pipeline:

Image title

Almost 40 times, it didn't work, but at the end, if you try to find what your problem is, search for answers, and you will definitely succeed.

Find out more about how Scalyr built a proprietary database that does not use text indexing for their log management tool.

Topics:
devops ,jenkins ,docker ,gitlab ,tutorial ,ci/cd

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}