DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Last call! Secure your stack and shape the future! Help dev teams across the globe navigate their software supply chain security challenges.

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Releasing software shouldn't be stressful or risky. Learn how to leverage progressive delivery techniques to ensure safer deployments.

Avoid machine learning mistakes and boost model performance! Discover key ML patterns, anti-patterns, data strategies, and more.

Related

  • Terraform Tips for Efficient Infrastructure Management
  • Migrating MuleSoft System API to AWS Lambda (Part 1)
  • Infrastructure as Code (IaC) Tools, Part 1: Overview of Tools
  • What Is API-First?

Trending

  • Unlocking AI Coding Assistants Part 2: Generating Code
  • Breaking Bottlenecks: Applying the Theory of Constraints to Software Development
  • Stateless vs Stateful Stream Processing With Kafka Streams and Apache Flink
  • Simplify Authorization in Ruby on Rails With the Power of Pundit Gem
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. Deployment
  4. How To Migrate Terraform State to GitLab CI/CD

How To Migrate Terraform State to GitLab CI/CD

Readers will migrate Terraform state to GitLab CI/CD, which is a GitOps best practice that allows multiple engineers to work together to develop infrastructure.

By 
Anthony Neto user avatar
Anthony Neto
·
Feb. 24, 23 · Tutorial
Likes (2)
Comment
Save
Tweet
Share
6.5K Views

Join the DZone community and get the full member experience.

Join For Free

As a software professional handling Infrastructure as Code (IaC), chances are you work a lot with Terraform. When helping new clients use IaC, it is common to simplify things, but managing a Terraform state file is the first challenge you face. Essentially, Terraform state contains sensitive information, which shouldn’t be stored by source control but, at the same time, won’t scale if you have multiple users working on the same Terraform state. The answer to that? Backends.

It is important to note that you could store that state file on an S3 bucket and use DynamoDB to manage the locking state. However, this approach will force you to create additional resources, which makes it a complicated option, especially if the client is using GitLab. GitLab recently lowered the entry barrier to integrating Terraform by providing a way to store and manage Terraform state, as well as an easy way to set up a CI around it.

In this article, we will explain what a Terraform state file is, how to migrate it to GitLab, and setting up a CI pipeline for it. You can visit our repository here.

Table of Contents

  • What Is Terraform State?
  • How To Get GitLab to Manage Terraform State
  • How To Get GitLab to Run Your IaC Through a CI Pipeline
    • Bonus Tip: Infracost
  • Conclusion

What Is Terraform State?

Terraform records any information about the infrastructure defined in your code via a state file. Written in JSON, it essentially records a mapping from the Terraform code to the real resources created. Below is an example of what a terraform.tfstate would look like. Primarily, every time you run Terraform it will fetch the latest status for its EC2 instance and compare it with your Terraform configuration to determine what changes need to be applied:

 
{   
     "version": 4,    
     "terraform_version": "0.12.0",    
     "serial": 1,    
     "lineage": "1f2087f9-4b3c-1b66-65db-8b78faafc6fb",    
     "outputs": {},    
     "resources": [     
      {        
         "mode": "managed",        
         "type": "aws_instance",       
         "name": "example",       
         "provider": "provider.aws",       
         "instances": [         
           {           
             "schema_version": 1,           
             "attributes": {              
               "ami": "ami-0c55b159cbfafe1f0",             
               "availability_zone": "us-west-2a",             
               "id": "i-00a123a0accffffff",            
               "instance_state": "running",             
               "instance_type": "t2.micro",            
               "(...)": "(truncated)"           
            }         
          }       
        ]      
      }   
    ]  
 }


By default, this terraform.tfstate is stored locally where you have your Terraform files, plan, and apply your changes. For a personal project where you are just running some tests; it’s fine, but not the recommended way, here’s why:

  • Stored in a shared location: if you were hosting this state file on your local workstation and had to work with another engineer, things would get complicated. Both of you will have to make sure you are using the latest version of the state and you could run into race conditions if you run a Terraform plan or apply at the same time.
  • Protect sensitive information: A generated state file can contain encryption keys and infrastructure passwords. However, state files aren’t encrypted by default, and storing sensitive information in plain text is a bad idea.
  • Locking: Most version control systems do not support any form of locking, which prevents two team members from running Terraform apply simultaneously on the same state file. This is another reason why we will not see a state file managed by source control.

How To Get GitLab to Manage Terraform State

With Terraform being considered the standard in cloud infrastructure provisioning, it has been a year or so since GitLab began to offer a way to store and manage your Terraform state. For this reason, we wanted to share the migration process with you as we recently started using GitLab to manage our IaC.

For this article, we presume that you are using a local state, and have your state managed with an AWS S3 Bucket or another backend solution. 

Firstly, you will need to change your backend.tf to use HTTP: 

 
terraform {  
    backend "http" {} 
  }


Next, you will need to set up four variables in your terminal:

1. PROJECT_ID: You can find this easily by navigating to your repo on the “Project Overview” page.

Backend

2. TF_USERNAME: The Gitlab username that has access to the repo you’re working on.

3. TF_PASSWORD: Access token generated from your GitLab user.

4. TF_ADDRESS: URL of the remote state backend.

 
PROJECT_ID="28450092"
TF_USERNAME="florianpialoux"
TF_PASSWORD="123456789"
TF_ADDRESS="https://gitlab.com/api/v4/projects/${PROJECT_ID}/terraform/state/aws-buckets"


You may now run the migration command that will move your Terraform state from its previous location to GitLab with the following command:

 
  terraform init \  
    -migrate-state \  
    -backend-config=address=${TF_ADDRESS} \  
    -backend-config=lock_address=${TF_ADDRESS}/lock \ 
    -backend-config=unlock_address=${TF_ADDRESS}/lock \  
    -backend-config=username=${TF_USERNAME} \ 
    -backend-config=password=${TF_PASSWORD} \  
    -backend-config=lock_method=POST \ 
    -backend-config=unlock_method=DELETE \ 
    -backend-config=retry_wait_min=5


You will need to provide a confirmation by a “yes” so that GitLab can start managing your state file. Here’s an example from a local state to GitLab:

Example S3 to GitLab:

S3

Now, you can navigate to Infrastructure > Terraform from the GitLab interface and see your state:
Gitlab Infrastructure Interface

I noticed for some of the state files I had from S3 will be blank even after using the migrate-state command ran previously. In this case, you can run this:  

 
terraform state pull > aws-buckets.json


Copy and paste the content from the S3 state and run a push:

 
terraform state push -lock=true aws-buckets.json


GitLab supports versioning for your Terraform state file but viewing/restoring older versions through the WebUI will require you to be using a GitLab Premium plan. If not, you will need to make a GraphQL API request.

How To Get GitLab to Run your IaC Through a CI Pipeline

GitLab provides a Docker image that contains GitLab-Terraform, which is a thin wrapper script around the official Terraform binary. Alternatively, you could use the official Docker image by Hashicorp. You can find more information about the GitLab Terraform Image here.

Once the Terraform apply job runs, you will be able to see when the state was used and with which pipeline.

Pipeline

Learn more about what our gitlab-ci.yml looks like here. Below, are the variables that will need to be defined on the project level.

Project Level

Bonus Tip: Infracost

As you might have noticed, looking at our gitlab-ci.yaml we added Infracost, which allows us to have more control over our cloud billing as it gives you a cost estimate whenever you define a new resource to your IaC.

Conclusion

Having your Terraform state and CI running on Gitlab is a great way to follow GitOps best practices. They both make a great combination to develop and deploy IaC. Since most of you might be already using GitLab for your repositories, it becomes much simpler to have your IaC under one roof and let GitLab manage your Terraform state by supporting encryption in transit and at rest, as well as versioning, locking, and unlocking the state.

API AWS Amazon DynamoDB GitLab Plain text Terraform (software) Continuous Integration/Deployment Data migration GraphQL Infrastructure as code JSON Cloud Pipeline (software) push Integration

Published at DZone with permission of Anthony Neto. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Terraform Tips for Efficient Infrastructure Management
  • Migrating MuleSoft System API to AWS Lambda (Part 1)
  • Infrastructure as Code (IaC) Tools, Part 1: Overview of Tools
  • What Is API-First?

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!