Pulumi: Modern Infrastructure as Code With Real Programming Languages
Pulumi is an Infrastructure as Code platform that lets developers and teams create, deploy, and manage cloud resources using familiar programming languages.
CORE
·
Join the DZone community and get the full member experience.
Join For FreeAfter a long journey with Terraform, when Terraform introduced HCL2, I started exploring for an alternative IaC tool to write code in my programming language of choice, and that's when I found Pulumi. Founded in 2017, Pulumi has emerged as a powerful alternative to traditional IaC tools by bridging the gap between software development and infrastructure management.
Pulumi is a modern Infrastructure as Code (IaC) platform that enables developers and infrastructure teams to create, deploy, and manage cloud resources using familiar programming languages instead of domain-specific languages (DSLs) or YAML templates.
What Is Pulumi?
Pulumi is an open-source Infrastructure as Code (IaC) tool that allows you to define cloud infrastructure using general-purpose programming languages such as TypeScript, JavaScript, Python, Go, C#, Java, and YAML. Instead of learning proprietary configuration languages, developers can leverage their existing programming skills to build, configure, and deploy infrastructure across multiple cloud providers, including AWS, Azure, Google Cloud Platform, IBM Cloud, Kubernetes, and many others.
The code snippets in this article will be in Python programming language.
Core Features of Pulumi
1. Multi-Language Support
Pulumi supports multiple programming languages, allowing teams to choose the language they're most comfortable with:
- TypeScript/JavaScript: Ideal for web developers and those familiar with Node.js
- Python: Perfect for data scientists, DevOps engineers, and Python developers
- Go: Great for system programmers and those building high-performance applications
- C#: Excellent for .NET developers and enterprise environments
- Java: Suitable for enterprise Java developers
- YAML: For those who prefer declarative configuration
2. Universal Cloud Support
Pulumi provides comprehensive support for 100+ cloud providers and services, enabling you to manage infrastructure across any platform using consistent APIs and workflows.
3. State Management
Pulumi automatically manages infrastructure state, tracking the current state of your resources and enabling safe updates, rollbacks, and collaborative workflows. State can be stored in:
- Pulumi Service (SaaS)
- Self-managed backends (S3, Azure Blob, GCS)
- Local filesystem
4. Policy as Code
Pulumi CrossGuard allows you to define and enforce policies across your infrastructure using the same programming languages, ensuring compliance and security standards are met automatically.
5. Secrets Management
Built-in secrets management encrypts sensitive configuration data like API keys, passwords, and certificates, ensuring they're never stored in plain text.
6. Testing and Validation
Since infrastructure is defined in real programming languages, you can write unit tests, integration tests, and use standard testing frameworks to validate your infrastructure code.
7. Rich Ecosystem
Pulumi integrates with existing development tools, including:
- CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)
- IDEs with full IntelliSense and debugging support
- Package managers (npm, pip, NuGet)
- Testing frameworks
Installation and Setup
Installing Pulumi
Prerequisites
- Python 3.7 or later
- pip (Python package manager)
Step 1: Install Pulumi CLI
macOS:
# Using Homebrew
brew install pulumi
# Using curl
curl -fsSL https://get.pulumi.com | shWindows:
# Using Chocolatey
choco install pulumi
# Using Windows installer
# Download from https://github.com/pulumi/pulumi/releasesLinux:
# Using curl
curl -fsSL https://get.pulumi.com | sh
# Add to PATH
export PATH=$PATH:$HOME/.pulumi/binStep 2: Verify Installation
pulumi versionStep 3: Install Python Dependencies
# Install pulumi Python SDK
pip install pulumi
# Install provider packages as needed
pip install pulumi-random
pip install pulumi-docker
# Add other providers as requiredCreating Your First Project
# Create a new directory
mkdir my-pulumi-project
cd my-pulumi-project
# Initialize a new Pulumi project
pulumi new python
# This creates the basic project structure
Check this Pulumi code repository for code samples.
Pulumi Project Structure
A typical Pulumi Python project has the following structure:
my-pulumi-project/
├── __main__.py # Main program entry point
├── Pulumi.yaml # Project definition file
├── Pulumi.dev.yaml # Stack configuration (dev stack)
├── requirements.txt # Python dependencies
├── venv/ # Python virtual environment (optional)
└── components/ # Reusable components (optional)
├── __init__.py
└── web_app.pyKey Files Explained
Pulumi.yaml
Project definition:
name: my-pulumi-project
runtime: python
description: A minimal Python Pulumi programmain.py
Main program file:
"""A Python Pulumi program"""
import pulumi
# Your infrastructure code goes here
pulumi.export("message", "Hello, Pulumi!")requirements.txt
Python dependencies:
pulumi>=3.0.0,<4.0.0
pulumi-randomGetting Started With Code Examples
Pulumi Terminology and Concepts
Understanding Pulumi's terminology is crucial for effective infrastructure management. Here's a comprehensive overview of key concepts and how they compare to Terraform:
Core Pulumi Concepts
Projects
A Project is a directory containing a Pulumi program and its metadata. It's defined by a Pulumi.yaml file and contains all the code that describes your infrastructure.
# Pulumi.yaml
name: my-web-app
runtime: python
description: A web application infrastructurePrograms
A Program is the code that defines your infrastructure resources. In Python, this is typically your __main__.py file or the entry point specified in your project configuration.
# __main__.py - This is your Pulumi program
import pulumi
import pulumi_random as random
# This program defines infrastructure resources
password = random.RandomPassword("app-password", length=16)
pulumi.export("password", pulumi.Output.secret(password.result))Stacks
A Stack is an isolated, independently configurable instance of a Pulumi program. Each stack has its own configuration, secrets, and state. You can have multiple stacks for the same program (e.g., dev, staging, production).
# Create and manage stacks
pulumi stack init dev
pulumi stack init staging
pulumi stack init production
# Switch between stacks
pulumi stack select devResources
Resources represent cloud infrastructure components like virtual machines, databases, or storage buckets. They are the fundamental building blocks in Pulumi.
import pulumi_random as random
# This creates a resource
pet_name = random.RandomPet("my-pet",
length=2,
separator="-"
)
# Resources have properties and methods
pulumi.export("pet_id", pet_name.id)State
State is Pulumi's record of which resources your program has created and their current properties. Pulumi automatically manages state for you.
Configuration
Configuration allows you to parameterize your Pulumi programs. Configuration values are stored per stack and can include secrets.
import pulumi
config = pulumi.Config()
environment = config.require("environment") # Required value
instance_count = config.get_int("instanceCount") or 3 # Optional with default
database_password = config.require_secret("dbPassword") # Secret valueOutputs
Outputs are values that are not known until your infrastructure is created. They represent properties of resources that are computed during deployment.
import pulumi
import pulumi_random as random
# Create a resource
password = random.RandomPassword("db-password", length=16)
# The 'result' is an Output - its value isn't known until deployment
pulumi.export("db_password", pulumi.Output.secret(password.result))
# You can transform outputs
formatted_password = password.result.apply(lambda pwd: f"Password: {pwd}")Component Resources
Component Resources are reusable infrastructure patterns that group multiple resources together into logical units.
class DatabaseCluster(pulumi.ComponentResource):
def __init__(self, name: str, args: dict, opts=None):
super().__init__("custom:DatabaseCluster", name, None, opts)
# Create multiple resources as part of this component
self.primary_password = random.RandomPassword(f"{name}-primary-pwd",
length=16, opts=pulumi.ResourceOptions(parent=self))
self.replica_password = random.RandomPassword(f"{name}-replica-pwd",
length=16, opts=pulumi.ResourceOptions(parent=self))
self.register_outputs({
"primary_password": self.primary_password.result,
"replica_password": self.replica_password.result
})
# Use the component
db_cluster = DatabaseCluster("my-cluster", {})Pulumi vs. Terraform Terminologies
| Concept | Pulumi | Terraform | Description |
|---|---|---|---|
| Project Organization | Project | Configuration | Top-level container for infrastructure code |
| Environment Isolation | Stack | Workspace | Isolated instances of the same infrastructure |
| Code Definition | Program | Configuration Files | The actual infrastructure code |
| Infrastructure Units | Resource | Resource | Individual cloud components |
| State Management | State | State File | Record of managed infrastructure |
| Dynamic Values | Output | Output | Values computed during deployment |
| Reusable Patterns | Component Resource | Module | Reusable infrastructure patterns |
| Configuration | Config | Variables | Runtime parameters |
| Secrets | Secret Config | Sensitive Variables | Encrypted configuration values |
Detailed Comparison
Projects vs. Configuration
# Pulumi Project Structure
my-app/
├── Pulumi.yaml # Project definition
├── __main__.py # Program code
└── requirements.txt # Dependencies# Terraform Configuration Structure
my-app/
├── main.tf # Main configuration
├── variables.tf # Input variables
├── outputs.tf # Output values
└── terraform.tf # Provider configurationStacks vs. Workspaces
Pulumi stacks:
# Each stack maintains separate state and config
pulumi stack init dev
pulumi config set environment development
pulumi config set instanceCount 2
pulumi stack init production
pulumi config set environment production
pulumi config set instanceCount 5# Access stack information in code
import pulumi
stack_name = pulumi.get_stack() # Returns current stack name
project_name = pulumi.get_project() # Returns project name
# Stack-aware resource naming
resource_name = f"{project_name}-{stack_name}-database"Terraform workspaces:
# Terraform workspaces
terraform workspace new dev
terraform workspace new production
terraform workspace select devConfiguration Management
Pulumi configuration:
# Set configuration per stack
pulumi config set app:name "MyApp"
pulumi config set app:instanceCount 3
pulumi config set --secret app:dbPassword "secretPassword123"
# Configuration files are created automatically
# Pulumi.dev.yaml, Pulumi.production.yaml, etc.# Use configuration in Python
import pulumi
config = pulumi.Config("app")
app_name = config.require("name")
instance_count = config.get_int("instanceCount") or 1
db_password = config.require_secret("dbPassword")
# Configuration can be objects too
config.get_object("tags") or {"environment": "dev"}Terraform variables:
# variables.tf
variable "app_name" {
description = "Application name"
type = string
}
variable "instance_count" {
description = "Number of instances"
type = number
default = 1
}
variable "db_password" {
description = "Database password"
type = string
sensitive = true
}State Management
Pulumi state:
- Automatically managed
- Multiple backend options (Pulumi Service, S3, Azure Blob, etc.)
- Built-in encryption
- No manual state file manipulation needed
# State is handled automatically
import pulumi_random as random
# This resource's state is automatically tracked
password = random.RandomPassword("my-password", length=16)
# No need to manage state files manuallyTerraform state:
- Requires explicit backend configuration
- Manual state file management sometimes needed
- State locking configuration required for teams
Advanced Pulumi Concepts
Transformations
Transformations allow you to modify resources automatically during creation:
def add_tags(args):
"""Add common tags to all resources"""
if hasattr(args, 'props') and 'tags' in args.props:
args.props['tags'] = {
**args.props.get('tags', {}),
'ManagedBy': 'Pulumi',
'Stack': pulumi.get_stack()
}
return pulumi.ResourceTransformationResult(args.props, args.opts)
# Apply transformation to all child resources
opts = pulumi.ResourceOptions(transformations=[add_tags])Aliases
Aliases help with resource refactoring and renaming:
# When renaming resources, use aliases to maintain continuity
old_resource = random.RandomPassword("new-name",
length=16,
opts=pulumi.ResourceOptions(
aliases=["urn:pulumi:stack::project::pulumi:providers:random::old-name"]
)
)This terminology section provides a comprehensive foundation for understanding Pulumi's concepts and how they relate to Terraform, making it easier for developers familiar with either tool to understand both platforms.
Code Examples
Example 1: Basic Resource Definition
import pulumi
import pulumi_random as random
# Create a random password
password = random.RandomPassword("db-password",
length=16,
special=True
)
# Create a random pet name
pet_name = random.RandomPet("my-pet",
length=2,
separator="-"
)
# Export outputs (password will be encrypted in state)
pulumi.export("db_password", pulumi.Output.secret(password.result))
pulumi.export("pet_name", pet_name.id)Example 2: Configuration and Conditional Logic
import pulumi
import pulumi_random as random
# Read configuration
config = pulumi.Config()
environment = config.require("environment")
enable_monitoring = config.get_bool("enableMonitoring") or False
instance_count = config.get_int("instanceCount") or 3
# Create resources based on environment
if environment == "production":
instance_size = "large"
replica_count = 5
elif environment == "staging":
instance_size = "medium"
replica_count = 3
else:
instance_size = "small"
replica_count = 1
# Generate random names for instances
instance_names = []
for i in range(instance_count):
name = random.RandomPet(f"instance-name-{i}",
length=2,
separator="-"
)
instance_names.append(name)
# Create tags for all resources
common_tags = {
"environment": environment,
"project": "my-project",
"managed-by": "pulumi"
}
# Export configuration and generated names
pulumi.export("environment", environment)
pulumi.export("instance_size", instance_size)
pulumi.export("replica_count", replica_count)
pulumi.export("instance_names", [name.id for name in instance_names])Example 3: Using Loops and Lists
import pulumi
import pulumi_random as random
config = pulumi.Config()
environments = config.get_object("environments") or ["dev", "staging", "prod"]
regions = config.get_object("regions") or ["us-east-1", "us-west-2"]
# Create resources for each environment and region combination
resources = {}
for env in environments:
resources[env] = {}
for region in regions:
# Create a unique identifier for this environment-region combo
identifier = random.RandomId(f"{env}-{region}-id",
byte_length=8
)
# Create a password for this combination
password = random.RandomPassword(f"{env}-{region}-password",
length=16,
special=True
)
resources[env][region] = {
"id": identifier.hex,
"password": password.result
}
# Export organized outputs
for env in environments:
for region in regions:
pulumi.export(f"{env}_{region}_id", resources[env][region]["id"])
pulumi.export(f"{env}_{region}_password",
pulumi.Output.secret(resources[env][region]["password"]))Example 4: Component Resources (Reusable Components)
import pulumi
import pulumi_random as random
class WebApplication(pulumi.ComponentResource):
"""A reusable web application component"""
def __init__(self, name: str, args: dict, opts: pulumi.ResourceOptions = None):
super().__init__("custom:WebApplication", name, None, opts)
# Create application identifier
self.app_id = random.RandomId(f"{name}-app-id",
byte_length=4,
opts=pulumi.ResourceOptions(parent=self)
)
# Create database password
self.db_password = random.RandomPassword(f"{name}-db-password",
length=16,
special=True,
opts=pulumi.ResourceOptions(parent=self)
)
# Create application configuration
self.config = {
"app_name": args.get("app_name", name),
"runtime": args.get("runtime", "python"),
"environment": args.get("environment", "development"),
"database_url": self.db_password.result.apply(
lambda pwd: f"postgresql://user:{pwd}@localhost:5432/{name}"
)
}
# Create monitoring if enabled
self.monitoring_enabled = args.get("enable_monitoring", False)
if self.monitoring_enabled:
self.monitoring_token = random.RandomPassword(f"{name}-monitoring-token",
length=32,
special=False,
opts=pulumi.ResourceOptions(parent=self)
)
# Register outputs
self.register_outputs({
"app_id": self.app_id.hex,
"config": self.config
})
# Use the component
web_app = WebApplication("my-web-app", {
"app_name": "hello-world",
"runtime": "python",
"environment": "production",
"enable_monitoring": True
})
# Create multiple applications
apps = {}
app_names = ["frontend", "backend", "api"]
for app_name in app_names:
apps[app_name] = WebApplication(f"{app_name}-app", {
"app_name": app_name,
"runtime": "python",
"environment": pulumi.get_stack(),
"enable_monitoring": app_name == "api" # Only enable monitoring for API
})
# Export application endpoints
pulumi.export("primary_app_id", web_app.app_id.hex)
for app_name, app in apps.items():
pulumi.export(f"{app_name}_app_id", app.app_id.hex)Example 5: Working With Transformations
import pulumi
import pulumi_random as random
def add_common_tags(args):
"""Transformation function to add common tags to all resources"""
if hasattr(args, 'props') and 'tags' in args.props:
# Add common tags
common_tags = {
"project": "my-project",
"owner": "dev-team",
"stack": pulumi.get_stack(),
"created-by": "pulumi"
}
# Merge with existing tags
existing_tags = args.props.get('tags', {})
args.props['tags'] = {**existing_tags, **common_tags}
return pulumi.ResourceTransformationResult(args.props, args.opts)
# Apply transformations to resources
transformation_opts = pulumi.ResourceOptions(
transformations=[add_common_tags]
)
# Create resources with automatic tag application
server_id = random.RandomId("web-server-id",
byte_length=8,
opts=transformation_opts
)
database_password = random.RandomPassword("database-password",
length=20,
special=True,
opts=transformation_opts
)
pulumi.export("server_id", server_id.hex)
pulumi.export("db_password", pulumi.Output.secret(database_password.result))Configuration Management
Set up configuration for your Pulumi projects:
# Set configuration values
pulumi config set environment development
pulumi config set instanceCount 3
pulumi config set --secret dbPassword mySecretPassword123
pulumi config set --path regions[0] us-east-1
pulumi config set --path regions[1] us-west-2
# List all configuration
pulumi config
# Get configuration in Python
config = pulumi.Config()
environment = config.require("environment")
instance_count = config.get_int("instanceCount") or 1
regions = config.get_object("regions") or ["us-east-1"]Testing Infrastructure Code
# test_infrastructure.py
import unittest
from unittest.mock import patch
import pulumi
class TestInfrastructure(unittest.TestCase):
def setUp(self):
# Set up Pulumi mocks
pulumi.runtime.set_mocks({
'newResource': self.new_resource_mock,
'call': self.call_mock
})
def new_resource_mock(self, args):
return {
'id': f"{args.name}_id",
'state': args.inputs
}
def call_mock(self, args):
return args.inputs
@patch('pulumi.Config')
def test_resource_creation(self, mock_config):
# Mock configuration
mock_config.return_value.require.return_value = "test"
mock_config.return_value.get_int.return_value = 2
# Import and test your infrastructure
import __main__
# Add your test assertions here
self.assertTrue(True) # Replace with actual tests
if __name__ == '__main__':
unittest.main()Running Your Pulumi Program
# Preview changes
pulumi preview
# Deploy infrastructure
pulumi up
# View stack outputs
pulumi stack output
# Update infrastructure
pulumi up
# Destroy infrastructure
pulumi destroy
Opening the link from the pulumi up output will show you the updates.
Pulumi vs. Other IaC Tools
Pulumi vs. Terraform
| Feature | Pulumi | Terraform |
|---|---|---|
| Language | Real programming languages (Python, TypeScript, Go, etc.) | HCL (HashiCorp Configuration Language) |
| Learning Curve | Leverages existing programming knowledge | Requires learning HCL syntax |
| Logic & Control Flow | Full programming constructs (loops, conditionals, functions) | Limited HCL constructs |
| Testing | Standard testing frameworks and practices | Limited testing capabilities |
| IDE Support | Full IntelliSense, debugging, refactoring | Basic syntax highlighting |
| State Management | Built-in with multiple backend options | Requires separate state configuration |
| Modularity | Native language packages and modules | Terraform modules |
| Community | Growing ecosystem | Large, mature ecosystem |
Pulumi vs. Ansible
| Feature | Pulumi | Ansible |
|---|---|---|
| Primary Use | Infrastructure provisioning | Configuration management |
| Approach | Declarative with imperative capabilities | Imperative playbooks |
| Language | Programming languages | YAML |
| State Tracking | Built-in state management | Stateless (idempotent) |
| Cloud Focus | Cloud-native infrastructure | Server configuration |
To understand what Ansible offers as a platform, check this article on DZone.
Use Cases for Pulumi
1. Multi-Cloud Deployments
Organizations seeking to avoid vendor lock-in can use Pulumi to deploy across multiple cloud providers using consistent tooling and practices.
2. Developer-Friendly Infrastructure
Teams with strong programming backgrounds can leverage existing skills instead of learning new DSLs, leading to faster adoption and reduced learning curves.
3. Complex Infrastructure Logic
When infrastructure requires complex logic, conditionals, or dynamic resource creation, Pulumi's programming language approach provides more flexibility than template-based tools.
4. Continuous Integration/Deployment
Pulumi integrates seamlessly with existing CI/CD pipelines, allowing infrastructure changes to be tested, reviewed, and deployed using standard software development practices.
5. Policy and Compliance
Organizations can implement policy as code using CrossGuard to ensure infrastructure compliance across all deployments automatically.
Best Practices
1. Project Organization
- Use separate projects for different environments (dev, staging, production).
- Implement proper naming conventions.
- Organize code into reusable components.
2. Configuration Management
- Use Pulumi's configuration system for environment-specific values.
- Store secrets securely using Pulumi's encryption.
- Avoid hardcoding values in infrastructure code.
3. Testing Strategy
- Write unit tests for infrastructure components.
- Implement integration tests for full deployments.
- Use property-based testing for complex scenarios.
4. State Management
- Choose an appropriate backend for your team size and requirements.
- Implement proper access controls for state files.
- Regular backups of state data.
5. Version Control
- Store infrastructure code in version control.
- Use branching strategies appropriate for your deployment model.
- Implement code review processes.
Conclusion
Pulumi represents a paradigm shift in infrastructure as code, bringing the power and flexibility of general-purpose programming languages to infrastructure management. By allowing developers to use familiar languages and tools, Pulumi reduces the barrier to entry for infrastructure automation while providing powerful features for complex deployments.
The choice between Pulumi and other IaC tools ultimately depends on your team's expertise, requirements, and preferences. Teams with strong programming backgrounds often find Pulumi's approach more intuitive, while those comfortable with declarative configuration languages might prefer traditional tools like Terraform or CloudFormation.
As cloud infrastructure continues to grow in complexity, tools like Pulumi that bridge the gap between software development and infrastructure management will become increasingly valuable for organizations seeking to implement efficient, scalable, and maintainable infrastructure automation.
Opinions expressed by DZone contributors are their own.
Comments