Terragrunt: Faster runs with the runner pool and provider caching

A run --all across a large Terragrunt estate spends its time in two places. One is real work: OpenTofu/Terraform planning and applying changes. The other is overhead: scheduling that leaves available concurrency unused, and every unit downloading the same providers over and over. Terragrunt 1.0 went after both, with a rebuilt scheduling engine called the runner pool and automatic provider caching for OpenTofu 1.10 and later.

Neither of these is news if you follow the release notes, but they're worth understanding on their own terms, because the speedup they deliver is not a flat percentage. It grows with the size and shape of your estate. This post explains how each one works and what, if anything, you need to do to benefit from them.

How Terragrunt schedules a run --all

When you run a command across many units:

terragrunt run --all plan

Terragrunt discovers the units under the working directory, reads their dependency and dependencies blocks, and builds a Directed Acyclic Graph (DAG) from them. That graph becomes a run queue: for plan and apply, dependencies run before the units that depend on them; for destroy, the order reverses.

Units run concurrently up to a limit you control with --parallelism, but the ordering constraint always holds. A unit will not start until everything it depends on has finished successfully.

That much has been true of Terragrunt for years. What changed is how aggressively Terragrunt fills the available concurrency while respecting the graph.

The runner pool

The old model: dependency groups

Before the runner pool, Terragrunt grouped units by dependency depth and ran the groups one after another. Units with no dependencies went into group 1, units that depended only on group 1 went into group 2, and so on:

Group 1
- Module ./vpc
- Module ./dns

Group 2
- Module ./database
- Module ./cache

Group 3
- Module ./app

The model is easy to reason about and easy to print in logs, but it has a structural flaw: a group only starts when the entire previous group has finished. If vpc takes eight minutes and dns takes thirty seconds, cache sits idle for seven and a half minutes even though its only dependency finished almost immediately. Every group waits for the slowest unit in the group before it.

Failures had the same coarse granularity. A single failing unit could take down the whole group boundary, failing units that had no dependency relationship to the one that broke.

The pool model

The Runner Pool RFC replaced groups with a queue and a pool. Every discovered unit goes into the queue with metadata about what blocks it. Units with no unfinished dependencies are ready; the rest are blocked. Terragrunt maintains a pool of runners sized by --parallelism, and any time a runner is free and a unit is ready, the unit starts.

When a unit finishes successfully, Terragrunt removes it from the blocked-by lists of its dependents, and any dependent whose list just became empty is immediately eligible to run. There is no group boundary to wait on. In the example above, cache starts the moment dns finishes, while vpc is still running.

Failure handling got more precise at the same time. When a unit fails, only its dependents (and their dependents, transitively) are marked as having a failed ancestor and skipped. Unrelated units keep running. If you prefer the old stop-everything behavior, --fail-fast gives it back:

terragrunt run --all --fail-fast apply

Why the gain scales with estate size

In the worst case, where every unit in a layer takes the same amount of time, the pool and the group model finish together. Real infrastructure estates don't look like that. Unit runtimes vary wildly (provisioning a DNS record and a database cluster does not require the same amount of work), and the more units and the more depth your graph has, the more idle time the group model accumulates at group boundaries. The pool eliminates exactly that idle time, which is why the improvement is proportional to the size of the estate: small stacks see a modest gain, while estates with hundreds of units and uneven runtimes see runs finish far sooner.

There is nothing to enable. The runner pool rolled out experimentally during the 0.x series and is the engine behind every run --all and run --graph in 1.x. The Run Queue docs cover the knobs for shaping it, including --parallelism, --queue-ignore-errors, and unit filtering.

Provider caching

Scheduling is only half the story. The other large cost at scale is provider downloads.

The problem

Without caching, every unit downloads its own copy of every provider it uses and unpacks it into its own .terraform directory. The numbers add up fast: the AWS provider archive (v6.50.0) is roughly 200MB, around 900MB unpacked. A project with 50 units spends about 10GB of bandwidth and 45GB of disk on what should have been a single 200MB download.

OpenTofu/Terraform has a built-in plugin cache (TF_PLUGIN_CACHE_DIR), but historically it was unsafe under the concurrency of a run --all: parallel processes corrupted each other's cache writes. That forced an unhappy choice between caching and concurrency.

Automatic provider caching with OpenTofu 1.10+

OpenTofu 1.10 made concurrent access to the plugin cache safe, and Terragrunt builds on that directly. When Terragrunt detects OpenTofu 1.10 or later, it automatically sets TF_PLUGIN_CACHE_DIR to a shared cache directory for every OpenTofu process it spawns. Each provider is downloaded once and reused by every unit. This is the Automatic Provider Cache Dir feature, and it is on by default. No flags, no configuration:

terragrunt run --all apply

The default cache location is $HOME/.cache/terragrunt/providers on Linux (honoring $XDG_CACHE_HOME if set), $HOME/Library/Caches/terragrunt/providers on macOS, and %LocalAppData%\terragrunt\providers on Windows. You can point it elsewhere:

terragrunt run --all apply --provider-cache-dir /mnt/fast-disk/providers

or opt out for a run:

terragrunt run --all apply --no-auto-provider-cache-dir

Two things to be precise about. This path requires OpenTofu 1.10 or later, and it works only with OpenTofu, not Terraform. If the requirements aren't met, the feature silently does nothing and you fall through to the behavior described next.

The provider cache server, for Terraform and older OpenTofu

If you run Terraform, or OpenTofu older than 1.10, Terragrunt's Provider Cache Server solves the same problem a different way. Terragrunt starts a local registry server, points each OpenTofu/Terraform process at it through a generated CLI config, and the server guarantees each provider is downloaded and stored exactly once, with units receiving symlinks into the shared cache. Because the server owns the cache, it handles the concurrency that the older built-in plugin cache could not.

It is off by default. Enable it with a flag:

terragrunt run --all --provider-cache apply

or an environment variable, which is the convenient form in CI:

TG_PROVIDER_CACHE=true terragrunt run --all apply

The server is also worth knowing about even on current OpenTofu. At very large scale, filesystem lock contention between OpenTofu processes synchronizing on the shared cache directory can become its own bottleneck, and the cache server avoids it. The performance docs describe when each approach wins.

Source downloads

Units also fetch module and unit sources repeatedly. The Content Addressable Store (CAS) deduplicates those fetches the same way the provider cache deduplicates provider downloads, storing fetched content by hash and serving repeat requests locally. It became the default in Terragrunt 1.1, and the Terragrunt 1.1 release post covers it in depth.

What to enable, by version

• OpenTofu >= 1.10, current Terragrunt: nothing. The runner pool and automatic provider caching are both defaults.
• Terraform, or OpenTofu < 1.10: add --provider-cache (or TG_PROVIDER_CACHE=1) to your run --all invocations. The runner pool still applies automatically.
• Either way: don't set TF_PLUGIN_CACHE_DIR yourself when using run --all with Terraform or pre-1.10 OpenTofu. That recreates the concurrent-corruption problem the cache server exists to avoid.

A few caveats for CI. Both caches are directories on the local filesystem, so on ephemeral runners they only pay off within a single job unless you persist the cache directory between jobs. And the cache server adds startup overhead, so for a single-unit terragrunt plan it can be a net negative; it earns its keep on run --all, where many units share the savings.

The wins compound in CI. Every plan on every pull request walks the same graph and fetches the same providers, so shaving minutes off a run --all shaves them off every pipeline run your team waits on. If you're orchestrating those runs with Gruntwork Pipelines, the scheduling and caching improvements apply to every plan and apply it triggers, with no pipeline changes.

Wrapping up

Two mechanisms carry the performance story: the runner pool starts every unit the moment its dependencies finish instead of waiting on group boundaries, and provider caching turns N identical downloads into one. On OpenTofu 1.10+ with a current version of Terragrunt, both are already working for you. On Terraform or older OpenTofu, one flag gets you the caching half. The Run Queue, Automatic Provider Cache Dir, and Provider Cache Server docs have the full details.

Improvements like the runner pool and provider caching come out of the work we do supporting large Terragrunt estates. If that's the problem you're solving, check out Terragrunt Scale.