Read our newest book, Fundamentals of DevOps and Software Delivery, for free!

Terraform: Up & Running, 3rd edition Early Release is now available!

Headshot of Yevgeniy Brikman

Yevgeniy Brikman

APR 7, 2022 | 14 min read
Update, September 28, 2022: The final version of Terraform: Up & Running, 3rd edition has been published! Grab your copy now!I’m excited to announce that the early release of Terraform: Up & Running, 3rd edition, is now available! The 2nd edition came out in 2019 and it's remarkable how much has changed since then: Terraform went through six major releases (0.13, 0.14, 0.15, 1.0, 1.1, and 1.2), most Terraform providers went through several major upgrades of their own (e.g., the AWS provider went from 2.0 to 3.0 and 4.0), and the Terraform community has continued to grow at a frantic pace, which has led to the emergence of many new best practices, tools, and modules.I’ve tried to capture all these new developments in the 3rd edition of the book, adding 100 pages (!) of new content on top of the 2nd edition, including two totally new chapters, plus major updates to all the existing chapters. To give you a preview of all this new content, I’ll do a quick walkthrough of the top 10 problems that have been fixed in Terraform over the last few years.This is a two part blog post series. In the first part of the series (this blog post), I’ll go into detail on the following 5 problems and their solutions, based on snippets from the 3rd edition of the book:
  • 1.Multiple regions, accounts, and clouds: multi-region replication, plus an example of using Terraform and Kubernetes to deploy Docker containers.
  • 2.Provider versioning: required_providers and the lock file.
  • 3.Secrets management: examples of using different types of secret management tools (e.g., Vault, KMS, etc) with Terraform.
  • 4.CI / CD security: examples of using OIDC and isolated workers to set up a secure CI / CD pipeline for Terraform.
  • 5.Module iteration: examples of using count and for_each to do loops and conditionals with module blocks.
In the second part of the series, which will come out when the final version of the 3rd edition is published (Update, September 28, 2022: the 3rd edition is now published, and the second part of the blog post series is available!), I’ll cover 5 more problems and solutions, including input validation, refactoring, static analysis, policy enforcement, and maturity. Grab a copy of the book to get full access to all of this content!

Just about all of the code examples in the 2nd edition of the book used a single region in a single account of a single cloud (AWS). But what if you wanted to deploy into multiple regions? Or multiple accounts? Or multiple clouds (e.g., AWS, GCP, and Azure)?

To answer these questions, the 3rd edition of the book includes a brand new chapter: Chapter 7, Working with Multiple Providers. This chapter shows how to work with multiple Terraform providers to deploy to multiple regions, multiple accounts, and multiple clouds.To deploy into multiple regions or accounts, you create multiple copies of the same provider, each configured differently, and each with a unique alias parameter. You then tell resources and modules which provider alias to use via the provider and providers parameters, respectively. To see this in action, here’s a code snippet from Chapter 7 that shows how to use multiple provider blocks with a mysql module to deploy MySQL with multi-region replication:
# Configure one provider to deploy into us-east-2 provider "aws" { region = "us-east-2" alias = "primary" }# Configure another provider to deploy into us-west-1 provider "aws" { region = "us-west-1" alias = "replica" }# Deploy the MySQL primary in us-east-2 module "mysql_primary" { source = "../../../../modules/data-stores/mysql" providers = { aws = aws.primary } db_name = var.db_name db_username = var.db_username db_password = var.db_password # Must be enabled to support replication backup_retention_period = 1 }# Deploy a MySQL replica in us-west-1 module "mysql_replica" { source = "../../../../modules/data-stores/mysql" providers = { aws = aws.replica } # Make this a replica of the primary replicate_source_db = module.mysql_primary.arn }
To deploy into multiple clouds, you create multiple copies of different providers. Readers of the first two editions of this book often asked for examples of how to work with multiple clouds (e.g., AWS, Azure, GCP), but I struggled to find an example where it was practical to do this in a single module. Here’s why:At a high level, there are two opposite extremes in how companies approach multi-cloud:
  • Transparent portability: With this approach, the idea is to try to use all the clouds as one unified computing platform, abstracting away all the differences between cloud providers to make it easier to migrate a workload from one cloud to the other.
  • Cloud native: With this approach, the idea is to try to use each cloud independently, leveraging its unique services as much as possible.
In my experience, the transparent portability approach is much less effective. It’s too focused on chasing the purported benefits of multi-cloud—avoiding vendor lock-in, reduced pricing, increased resiliency—that, as I discuss in Chapter 7, are mostly myths, and don’t pay off for most companies. Even more importantly, the desire for transparent portability vastly underestimates the significant costs of trying to paper over the differences between clouds: the offerings from each cloud may look superficially similar—e.g., they all offer virtual machines—but under the hood, there are many differences, including significant variation in the mechanics of authentication, authorization, networking, data storage, replication, partitioning, secrets management, compliance, security model, performance, latency, availability, scalability, limits/throttles, support, and much else.Therefore, except for a few niche cases, I recommend the cloud native approach. This is also the approach that Terraform is designed for: you can use Terraform with multiple clouds, but you have to write separate code for each cloud, using the providers and resources native to that cloud. Therefore, even for multi-cloud deployments, it’s unusual to build a single Terraform module that deploys into multiple clouds (that is, uses multiple different providers in one module); it’s much more common to keep the code for each cloud in separate modules.So, instead of adding an unrealistic multi-cloud example, the 3rd edition of the book includes an example of how to use multiple different providers together in a more realistic scenario: namely, how to use the AWS provider with the Kubernetes provider to deploy Dockerized apps on Amazon EKS.Chapter 7 includes a crash course on Docker, Kubernetes, and EKS, and by the end of the chapter, you will have built several modules that allow you to use the following simple code to spin up an EKS cluster and deploy a web app into it:
# Use the AWS provider to deploy into the AWS us-east-2 region provider "aws" { region = "us-east-2" }# Use the Kubernetes provider to deploy into an EKS cluster provider "kubernetes" { host = module.eks_cluster.cluster_endpoint cluster_ca_certificate = base64decode( module.eks_cluster.cluster_certificate_authority[0].data ) token = data.aws_eks_cluster_auth.cluster.token }data "aws_eks_cluster_auth" "cluster" { name = module.eks_cluster.cluster_name }# Deploy an EKS cluster module "eks_cluster" { source = "../../modules/services/eks-cluster" name = "example-eks-cluster" min_size = 1 max_size = 2 desired_size = 1 instance_types = ["t3.small"] }# Deploy a simple web app into the EKS cluster using a # Kubernetes Deployment module "simple_webapp" { source = "../../modules/services/k8s-app" name = "simple-webapp" image = "training/webapp" replicas = 2 container_port = 5000 depends_on = [module.eks_cluster] }

Terraform providers change all the time, and sometimes in backward incompatible ways. How do you ensure that all of your team members and your CI servers are using the same provider versions everywhere?

To solve this problem, Terraform 0.13 introduced the required_providers block and Terraform 0.14 introduced the lock file, both of which are now covered in Chapter 8, Production Grade Terraform Code. The required_providers block allows you to specify which providers your code depends on, where to download the code for that provider, and the version constraints to enforce:
terraform { required_providers { # Use any 4.x version of the AWS provider aws = { source = "hashicorp/aws" version = "~> 4.0" } # Use the Kubernetes provider at any version at or above 2.7 kubernetes = { source = "hashicorp/kubernetes" version = ">= 2.7.0" } } }
The first time you run terraform init, Terraform will download the provider code you’ve specified in the required_providers block and record the exact versions it downloaded in a .terraform.lock.hcl file:
provider "registry.terraform.io/hashicorp/aws" { version = "4.2.0" constraints = "~> 4.0" hashes = [ "h1:7xPC2b+Plr514HPRf837t3YFzlSSIY03StrScaVIfw0=", "h1:N5oVH/WT+1U3/hfpqs2iQ6wkoK+1qrPbYZJ+6Ptx6a0=", "h1:qfnMtwFbsVJWvzxUCajm4zUkjEH9GDdT3FFYffEEhYQ=", "h1:s29zqs8kRCi7hFmF2IWn0OAsVrh+oEyOzB1W/aeeQ3I=" ] }provider "registry.terraform.io/hashicorp/kubernetes" { version = "2.8.0" constraints = "~> 2.0" hashes = [ "h1:LZLKGKTlBmG8jtMBdZ4ZMe+r15OQLSMYV0DktfROk+Y=", "h1:UZCCMTH49ziz6YDV5oCCoOHypOxZWvzc59IfZxVdWeI=", "h1:aU9axQagkwAmDUVqRY71UU1kgjBrgFKQYgpAhqQnOEk=", "h1:tfU8BStZIt2d6KIGTRNjWb09zeVzh3UFGNRGVgFce+A=" ] }
If you check this lock file into version control, any team member or CI server that runs init will end up downloading the exact same versions of the provider code, so there’s no chance of pulling in newer (possibly backward incompatible) versions by accident. Chapter 8 also includes new examples of how to use tools like tfenv and tgswitch to manage Terraform and Terragrunt versions.

When working with Terraform code, you often have to manage secrets, such as API keys and database passwords. How do you handle these secrets without storing them in plain text or leaking sensitive data into your logs?

The 3rd edition contains an entirely new chapter to answer this question: Chapter 6, Managing Secrets with Terraform. This chapter includes an overview of secrets management and a comparison of a variety of secret management tools:
Chapter 6 also has examples of how to use several types of secret stores with Terraform, including how to pass secrets from a personal secrets manager such as 1Password to Terraform, how to read secrets from files encrypted with KMS, and the example below, which shows how to read database credentials from AWS Secrets Manager:
# Read database credentials from AWS Secrets Manager data "aws_secretsmanager_secret_version" "creds" { secret_id = "db-creds" }locals { # The credentials are stored in JSON, so we need to decode them db_creds = jsondecode( data.aws_secretsmanager_secret_version.creds.secret_string ) }resource "aws_db_instance" "example" { identifier_prefix = "terraform-up-and-running" engine = "mysql" allocated_storage = 10 instance_class = "db.t2.micro" skip_final_snapshot = true db_name = "example-db" # Pass the secrets to the database username = local.db_creds.username password = local.db_creds.password }
The entire book has also been updated to mark all input and output variables that could contain secrets with the new sensitive keyword, which was introduced in Terraform 0.14 and 0.15 to tell Terraform to never log these values, as they may contain sensitive data:
variable "db_username" { description = "The username for the database" type = string sensitive = true }variable "db_password" { description = "The password for the database" type = string sensitive = true }

In order to effectively use Terraform as a team, you need to set up a CI / CD pipeline that runs plan and apply automatically on your code changes. To deploy arbitrary Terraform changes, your CI server needs arbitrary permissions—which is just another term for admin permissions. And that’s a big problem, because now you’re mixing the following ingredients all in one place:
  • 1.The CI server is typically accessible to every dev on your team.
  • 2.CI servers are designed to execute arbitrary code.
  • 3.The CI server now has admin permissions.
No matter how you slice it, this is a bad combination. In effect, what you’ve done is given every one of your developers admin permissions, plus exposed admin permissions to any snippet of code that happens to run on that server. Attackers are aware of this and are starting to target CI servers in particular (e.g., see 10 real-world stories of how we’ve compromised CI/CD pipelines).

There are several ingredients to setting up a secure CI / CD pipeline for Terraform. The first ingredient is to handle credentials on your CI server securely. The 3rd edition of the book adds examples of using environment variables, IAM roles, and arguably the most secure option of all, OpenID Connect (OIDC). Chapter 6 includes an example of using OIDC with GitHub Actions to authenticate to AWS, via an IAM role, without having to manage any credentials at all:
# Authenticate to AWS using OIDC - uses: aws-actions/configure-aws-credentials@v1 with: # Specify the IAM role to assume here role-to-assume: arn:aws:iam::1234567890:role/example-role aws-region: us-east-2# Run Terraform using HashiCorp's setup-terraform Action - uses: hashicorp/setup-terraform@v1 with: terraform_version: 1.1.0 terraform_wrapper: false run: | terraform init terraform apply -auto-appro
The second ingredient is to strictly limit what the CI server can do once it has authenticated: for example, in the OIDC snippet above, you’ll want to severely limit the permissions in that IAM role. But then how do you handle the admin permissions you need to deploy arbitrary Terraform changes?The answer is that instead of giving the CI server admin permissions, you isolate the admin permissions to a totally separate worker: e.g., a separate server, a separate container, etc. That worker should be extremely locked down, so no developers have access to it at all, and the only thing it allows is for the CI server to trigger that worker via an API that is designed to be extremely limited. For example, the API your worker exposes might only allow you to run specific commands (e.g., terraform plan and terraform apply), in specific repos (e.g., your live repo), in specific branches (e.g., the main branch), and so on.

A diagram of the Gruntwork Pipelines architecture, which uses ECS Fargate as an isolated worker.

This way, even if an attacker gets access to your CI server, they still won’t have access to the admin credentials, and all they can do is request a deployment on some code that’s already in your version control system. This is still not great, but it’s not nearly as much of a catastrophe as leaking the admin credentials fully. Check out Gruntwork Pipelines for an example implementation of both the OIDC and admin worker approach.

You built a module and you want to use it several times—in a loop, essentially—without having to copy and paste the code. However, Terraform 0.12 and below didn’t support count or for_each on module.

Terraform 0.13 added support for using count and for_each on module blocks, so Chapter 5, Terraform Tips and Tricks, has been updated to show you how to use loops and conditional logic with your modules.For example, if you have an iam-user module that can create a single IAM user, you can use for_each on a module block to create 3 IAM users as follows:
locals { user_names = ["neo", "trinity", "morpheus"] }module "users" { source = "../../../modules/landing-zone/iam-user" for_each = toset(local.user_names) user_name = each.key }

You’ve now had a small taste of just 5 of the problems that have been solved in the Terraform world in the last few years and are now covered by the 3rd edition of Terraform: Up & Running, including how to work with multiple regions, accounts, and clouds, how to control your provider versions, how to manage secrets securely with Terraform, how to set up a secure CI / CD pipeline, and how to do control logic with modules.In part 2 of this series I’ll cover 5 more problems and solutions, including:
  • 1.Validation. How to use validation, precondition, and postcondition blocks to perform checks before and after deployment.
  • 2.Refactoring. How to use moved blocks to safely refactor your Terraform code without having to do state surgery manually.
  • 3.Static analysis: How to perform other types of automated testing on Terraform code, such as static analysis.
  • 4.Policy enforcement: How to enforce company policies and compliance requirements using tools such as Terratest, OPA, and Sentinel.
  • 5.Maturity. How Terraform has become more stable due to the Terraform 1.0 release, the growth of the community, and the HashiCorp IPO.
Dive into all the details in part 2 of the series now. And if you enjoyed this content and want to go deeper, grab yourself a copy of Terraform: Up & Running, 3rd edition, and let me know what you think!
Share
Grunty
Resources

Explore our latest blog

Get the most up-to-date information and trends from our DevOps community.
TerraformResouces Image

Promotion Workflows with Terraform

How to configure GitOps-driven, immutable infrastructure workflows for Terraform using Gruntwork Patcher.
avatar

Jason Griffin

October 3, 2023 7 min read
TerraformResouces Image

The Impact of the HashiCorp License Change on Gruntwork Customers

How to configure GitOps-driven, immutable infrastructure workflows for Terraform using Gruntwork Patcher.
avatar

Josh Padnick

October 3, 2023 7 min read