v0.89.0+ Runner Pool: run_cmd() outputs are cached before the unit runs

[ferplnat]

Describe the bug

Now that runner pool is the model, when switching to v0.89.0+, I started encountering failures due to the following behavior.

run_cmd() outputs are cached before the unit is processed. This is different from the "group" behavior in that run_cmd() outputs are calculated when the unit is processed.

Steps To Reproduce

git init
mkdir unit
mkdir -p live/test
touch unit/main.tf

# unit/terragrunt.hcl
include "root" {
  path = find_in_parent_folders("root.hcl")
}

# live/root.hcl
inputs = {
  # Test scenario for brevity
  test = run_cmd("pwgen", "-s", "25", "1")
}

# live/test/terragrunt.stack.hcl
unit "test1" {
  source = "${get_repo_root()}//unit"
  path = "test1"
}

unit "test2" {
  source = "${get_repo_root()}//unit"
  path = "test2"
}

cd live
terragrunt --log-level=debug run --all validate

Observe debug logs to see run_cmd() caching before the units are run.

Expected behavior

run_cmd() when not using --terragrunt-global-cache should be processed at unit processing time and not beforehand.

Nice to haves

v0.88.1 output

terragrunt --no-color --log-level=debug run --all validate

17:29:22.650 DEBUG  Terragrunt Version: 0.88.1
17:29:22.650 DEBUG  Searching for stack files in .
17:29:22.651 DEBUG  Found stack file ./test/terragrunt.stack.hcl
17:29:22.651 DEBUG  Generating stack level 0 with 1 files
17:29:22.651 DEBUG  Reading Terragrunt stack config file at ./test/terragrunt.stack.hcl
17:29:22.652 DEBUG  Running command: git rev-parse --show-toplevel
17:29:22.652 DEBUG  Command output will be suppressed.
17:29:22.670 DEBUG  git show-toplevel result: 


..

17:29:22.670 INFO   Generating unit test2 from ./test/terragrunt.stack.hcl
17:29:22.671 DEBUG  Generating: test2 (..//unit) to ./test/.terragrunt-stack/test2
17:29:22.670 INFO   Generating unit test1 from ./test/terragrunt.stack.hcl
17:29:22.671 DEBUG  Generating: test1 (..//unit) to ./test/.terragrunt-stack/test1
17:29:22.671 DEBUG  No values to write in ./test/.terragrunt-stack/test2
17:29:22.671 DEBUG  No values to write in ./test/.terragrunt-stack/test1
17:29:22.671 DEBUG  Searching for stack files in .
17:29:22.671 DEBUG  Found stack file ./test/terragrunt.stack.hcl
17:29:22.672 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.672 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.672 DEBUG  [test/.terragrunt-stack/test1] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.672 DEBUG  [test/.terragrunt-stack/test1] Setting download directory for unit ./test/.terragrunt-stack/test1 to ./test/.terragrunt-stack/test1/.terragrunt-cache
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] Setting download directory for unit ./test/.terragrunt-stack/test2 to ./test/.terragrunt-stack/test2/.terragrunt-cache
17:29:22.673 DEBUG  Stack at ../live:
  => Unit ./test/.terragrunt-stack/test1 (excluded: false, assume applied: false, dependencies: [])
  => Unit ./test/.terragrunt-stack/test2 (excluded: false, assume applied: false, dependencies: [])
17:29:22.673 INFO   The runner at . will be processed in the following order for command validate:
Group 1
- Unit ./test/.terragrunt-stack/test1
- Unit ./test/.terragrunt-stack/test2


17:29:22.673 DEBUG  [test/.terragrunt-stack/test1] Unit ./test/.terragrunt-stack/test1 must wait for 0 dependencies to finish
17:29:22.673 DEBUG  [test/.terragrunt-stack/test1] Running ./test/.terragrunt-stack/test1
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] Unit ./test/.terragrunt-stack/test2 must wait for 0 dependencies to finish
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] Running ./test/.terragrunt-stack/test2
17:29:22.673 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.673 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.673 DEBUG  [test/.terragrunt-stack/test1] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.673 DEBUG  [test/.terragrunt-stack/test2] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.674 DEBUG  [test/.terragrunt-stack/test1] using cache key for version files: r01AJjVD7VSXCQk1ORuh_no_NRY
17:29:22.674 DEBUG  [test/.terragrunt-stack/test2] using cache key for version files: r01AJjVD7VSXCQk1ORuh_no_NRY
17:29:22.674 DEBUG  [test/.terragrunt-stack/test1] Running command: terraform -version
17:29:22.674 DEBUG  [test/.terragrunt-stack/test1] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test1
17:29:22.674 DEBUG  [test/.terragrunt-stack/test2] Running command: terraform -version
17:29:22.674 DEBUG  [test/.terragrunt-stack/test2] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test2
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] terraform version: 1.9.2
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] terraform version: 1.9.2
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] Reading Terragrunt config file at ./test/.terragrunt-stack/test1/terragrunt.hcl
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] Reading Terragrunt config file at ./test/.terragrunt-stack/test2/terragrunt.hcl
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test2] Included config ./root.hcl has strategy shallow merge: merging config in (shallow) for dependency.
17:29:22.704 DEBUG  [test/.terragrunt-stack/test1] Included config ./root.hcl has strategy shallow merge: merging config in (shallow) for dependency.
17:29:22.705 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.705 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.705 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:29:22.705 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:29:22.705 DEBUG  [test/.terragrunt-stack/test2] Running command: pwgen -s 25 1
17:29:22.705 DEBUG  [test/.terragrunt-stack/test1] Running command: pwgen -s 25 1
XfyPDvOa3sgreXeGzXzY36Qft
ksxOO0VnunUmeOcfSl97Xkj1G
17:29:22.707 DEBUG  [test/.terragrunt-stack/test2] run_cmd output: [XfyPDvOa3sgreXeGzXzY36Qft]
17:29:22.707 DEBUG  [test/.terragrunt-stack/test1] run_cmd output: [ksxOO0VnunUmeOcfSl97Xkj1G]
17:29:22.707 DEBUG  [test/.terragrunt-stack/test2] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.707 DEBUG  [test/.terragrunt-stack/test1] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:29:22.707 DEBUG  [test/.terragrunt-stack/test2] Running command: terraform init -no-color
17:29:22.707 DEBUG  [test/.terragrunt-stack/test2] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test2
17:29:22.707 DEBUG  [test/.terragrunt-stack/test1] Running command: terraform init -no-color
17:29:22.707 DEBUG  [test/.terragrunt-stack/test1] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test1
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: Initializing the backend...
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: Initializing the backend...
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: Initializing provider plugins...
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: Terraform has been successfully initialized!
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: Initializing provider plugins...
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: any changes that are required for your infrastructure. All Terraform commands
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: Terraform has been successfully initialized!
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: should now work.
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: If you ever set or change modules or backend configuration for Terraform,
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: rerun this command to reinitialize your working directory. If you forget, other
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: any changes that are required for your infrastructure. All Terraform commands
17:29:22.732 INFO   [test/.terragrunt-stack/test1] terraform: commands will detect it and remind you to do so if necessary.
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: should now work.
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: If you ever set or change modules or backend configuration for Terraform,
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: rerun this command to reinitialize your working directory. If you forget, other
17:29:22.732 INFO   [test/.terragrunt-stack/test2] terraform: commands will detect it and remind you to do so if necessary.
17:29:22.733 DEBUG  [test/.terragrunt-stack/test2] Running command: terraform validate -no-color
17:29:22.733 DEBUG  [test/.terragrunt-stack/test2] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test2
17:29:22.733 DEBUG  [test/.terragrunt-stack/test1] Running command: terraform validate -no-color
17:29:22.734 DEBUG  [test/.terragrunt-stack/test1] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test1
17:29:22.758 STDOUT [test/.terragrunt-stack/test1] terraform: Success! The configuration is valid.
17:29:22.758 STDOUT [test/.terragrunt-stack/test2] terraform: Success! The configuration is valid.
17:29:22.760 DEBUG  [test/.terragrunt-stack/test2] Unit ./test/.terragrunt-stack/test2 has finished successfully!
17:29:22.760 DEBUG  [test/.terragrunt-stack/test1] Unit ./test/.terragrunt-stack/test1 has finished successfully!

v0.89.0 output

terragrunt --no-color --log-level=debug run --all validate

17:32:09.972 DEBUG  Terragrunt Version: 0.89.0
17:32:09.972 DEBUG  Running command: terraform -version
17:32:09.972 DEBUG  Engine is not enabled, running command directly in .
17:32:09.999 DEBUG  terraform version: 1.9.2
17:32:09.999 DEBUG  Auto provider cache dir setup failed: auto provider cache dir requires OpenTofu, but detected terraform
17:32:09.999 DEBUG  Searching for stack files in .
17:32:09.999 DEBUG  Found stack file ./test/terragrunt.stack.hcl
17:32:09.999 DEBUG  Generating stack level 0 with 1 files
17:32:09.999 DEBUG  Reading Terragrunt stack config file at ./test/terragrunt.stack.hcl
17:32:10.000 DEBUG  Running command: git rev-parse --show-toplevel
17:32:10.000 DEBUG  Command output will be suppressed.
17:32:10.010 DEBUG  git show-toplevel result: 


..

17:32:10.010 INFO   Generating unit test2 from ./test/terragrunt.stack.hcl
17:32:10.010 DEBUG  Generating: test2 (..//unit) to ./test/.terragrunt-stack/test2
17:32:10.010 INFO   Generating unit test1 from ./test/terragrunt.stack.hcl
17:32:10.010 DEBUG  Generating: test1 (..//unit) to ./test/.terragrunt-stack/test1
17:32:10.011 DEBUG  No values to write in ./test/.terragrunt-stack/test2
17:32:10.011 DEBUG  No values to write in ./test/.terragrunt-stack/test1
17:32:10.011 DEBUG  Searching for stack files in .
17:32:10.011 DEBUG  Found stack file ./test/terragrunt.stack.hcl
17:32:10.011 INFO   Using runner pool for stack .
17:32:10.011 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.011 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.011 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.012 DEBUG  [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Included config ./root.hcl has strategy shallow merge: merging config in (shallow) for dependency.
17:32:10.012 DEBUG  Included config ./root.hcl has strategy shallow merge: merging config in (shallow) for dependency.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Did not find any locals block: skipping evaluation.
17:32:10.012 DEBUG  Running command: pwgen -s 25 1
17:32:10.012 DEBUG  Running command: pwgen -s 25 1
17:32:10.014 DEBUG  run_cmd output: [OoiSYIDkF7k54676dz08zWZbK]
17:32:10.014 DEBUG  Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.014 DEBUG  run_cmd output: [ZsrYjmzsNIqa9lNdwGCcsmY35]
17:32:10.014 DEBUG  Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.014 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] Setting download directory for unit ./test/.terragrunt-stack/test2 to ./test/.terragrunt-stack/test2/.terragrunt-cache
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] Setting download directory for unit ./test/.terragrunt-stack/test1 to ./test/.terragrunt-stack/test1/.terragrunt-cache
17:32:10.015 DEBUG  Stack at ../live:
  => Unit ./test/.terragrunt-stack/test1 (excluded: false, assume applied: false, dependencies: [])
  => Unit ./test/.terragrunt-stack/test2 (excluded: false, assume applied: false, dependencies: [])
17:32:10.015 INFO   The runner-pool runner at . will be processed in the following order for command validate:
- Unit ./test/.terragrunt-stack/test1
- Unit ./test/.terragrunt-stack/test2

17:32:10.015 DEBUG  Runner Pool Controller: starting with 2 tasks, concurrency 2147483647
17:32:10.015 DEBUG  Runner Pool Controller: found 2 readyEntries tasks
17:32:10.015 DEBUG  Runner Pool Controller: running ./test/.terragrunt-stack/test1
17:32:10.015 DEBUG  Runner Pool Controller: running ./test/.terragrunt-stack/test2
17:32:10.015 DEBUG  Runner Pool Controller: found 0 readyEntries tasks
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] Running ./test/.terragrunt-stack/test1
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] Running ./test/.terragrunt-stack/test2
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] [Partial] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] using cache key for version files: r01AJjVD7VSXCQk1ORuh_no_NRY
17:32:10.015 DEBUG  [test/.terragrunt-stack/test2] using cache key for version files: r01AJjVD7VSXCQk1ORuh_no_NRY
17:32:10.015 DEBUG  [test/.terragrunt-stack/test1] Running command: terraform -version
17:32:10.016 DEBUG  [test/.terragrunt-stack/test1] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test1
17:32:10.016 DEBUG  [test/.terragrunt-stack/test2] Running command: terraform -version
17:32:10.016 DEBUG  [test/.terragrunt-stack/test2] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test2
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] terraform version: 1.9.2
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] Reading Terragrunt config file at ./test/.terragrunt-stack/test1/terragrunt.hcl
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] terraform version: 1.9.2
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] Reading Terragrunt config file at ./test/.terragrunt-stack/test2/terragrunt.hcl
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] Included config ./root.hcl has strategy shallow merge: merging config in (shallow) for dependency.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] Included config ./root.hcl has strategy shallow merge: merging config in (shallow) for dependency.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] Did not find any locals block: skipping evaluation.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] Did not find any locals block: skipping evaluation.
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] run_cmd, cached output: [ZsrYjmzsNIqa9lNdwGCcsmY35]
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] run_cmd, cached output: [OoiSYIDkF7k54676dz08zWZbK]
17:32:10.043 DEBUG  [test/.terragrunt-stack/test1] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.043 DEBUG  [test/.terragrunt-stack/test2] Included config ./root.hcl has strategy shallow merge: merging config in (shallow).
17:32:10.044 DEBUG  [test/.terragrunt-stack/test1] Running command: terraform init -no-color
17:32:10.044 DEBUG  [test/.terragrunt-stack/test1] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test1
17:32:10.044 DEBUG  [test/.terragrunt-stack/test2] Running command: terraform init -no-color
17:32:10.044 DEBUG  [test/.terragrunt-stack/test2] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test2
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: Initializing the backend...
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: Initializing the backend...
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: Initializing provider plugins...
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: Terraform has been successfully initialized!
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: any changes that are required for your infrastructure. All Terraform commands
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: should now work.
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: If you ever set or change modules or backend configuration for Terraform,
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: rerun this command to reinitialize your working directory. If you forget, other
17:32:10.069 INFO   [test/.terragrunt-stack/test2] terraform: commands will detect it and remind you to do so if necessary.
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: Initializing provider plugins...
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: Terraform has been successfully initialized!
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: You may now begin working with Terraform. Try running "terraform plan" to see
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: any changes that are required for your infrastructure. All Terraform commands
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: should now work.
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: If you ever set or change modules or backend configuration for Terraform,
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: rerun this command to reinitialize your working directory. If you forget, other
17:32:10.069 INFO   [test/.terragrunt-stack/test1] terraform: commands will detect it and remind you to do so if necessary.
17:32:10.071 DEBUG  [test/.terragrunt-stack/test2] Running command: terraform validate -no-color
17:32:10.071 DEBUG  [test/.terragrunt-stack/test2] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test2
17:32:10.071 DEBUG  [test/.terragrunt-stack/test1] Running command: terraform validate -no-color
17:32:10.071 DEBUG  [test/.terragrunt-stack/test1] Engine is not enabled, running command directly in ./test/.terragrunt-stack/test1
17:32:10.097 STDOUT [test/.terragrunt-stack/test2] terraform: Success! The configuration is valid.
17:32:10.097 STDOUT [test/.terragrunt-stack/test1] terraform: Success! The configuration is valid.
17:32:10.099 DEBUG  Runner Pool Controller: ./test/.terragrunt-stack/test1 succeeded
17:32:10.099 DEBUG  Runner Pool Controller: ./test/.terragrunt-stack/test2 succeeded
17:32:10.099 DEBUG  Runner Pool Controller: found 0 readyEntries tasks

❯❯ Run Summary  2 units  83ms
   ────────────────────────────
   Succeeded    2

Tested the same behavior on v0.90.0

Versions

• Terragrunt version: v0.89.0+
• OpenTofu/Terraform version: Terraform 1.9.2 (happens on any version)
• Environment details (Ubuntu 20.04, Windows 10, etc.): macOS, Ubuntu (GH Actions)

Additional context

Why is this impacting? Consider the following scenario:

GitHub Actions OIDC provides tokens with a validity of approximately 5 minutes. Because of this, setting TG_IAM_ASSUME_ROLE_WEB_IDENTITY_TOKEN does not work for long running deployments (e.g. run --all across 100+ units)

To circumvent this, a helper script is used to retrieve the GitHub Actions OIDC token at unit processing time, which looks like this:

• TG_IAM_ASSUME_ROLE_SESSION_NAME is set by the action globally
• iam_role is determined by including something like read_terragrunt_config(find_in_parent_folders("iam.hcl")).locals[values.environment]'
• iam_web_identity_token = run_cmd("--terragrunt-quiet", "path/to/oidc_script.sh") for runtime token retrieval.

If there's a different pattern to be used, I'm open to it. TG_AUTH_PROVIDER_CMD does not allow partial configurations, as only the awsRole.webIdentityToken would be known at the time that runs, not the role ARN.

[denis256]

Hi,
this caching of run_cmd is now intentional in Terragrunt v0.89.0 and later.

With the runner-pool model, Terragrunt performs configuration discovery and evaluation (including run_cmd() execution) up front, before the individual units are dispatched to runners. This change was necessary to allow consistent dependency resolution, deterministic evaluation, and full parallel scheduling of units across the runner pool.

If you need to execute dynamic commands (such as retrieving a short-lived OIDC token) at runtime rather than during discovery, you can achieve that using Terragrunt hooks.

Hooks will allow you to run arbitrary scripts before/after Terraform/OpenTofu commands for each unit.

terraform {
  before_hook "oidc_refresh" {
    commands = ["init", "plan", "apply"]
    execute  = ["bash", "path/to/oidc_script.sh"]
  }
}

https://terragrunt.gruntwork.io/docs/features/hooks/

[ferplnat]

I wasn't able to get a pattern working with before_hook that is able to leverage the built-in terragrunt AWS assumeRoleWithWebIdentity logic.

Does auth logic have to be completely handled by the before_hook script? If so -- that is a bummer.

It would be nice to have a pattern to keep the webIdentityToken fresh throughout a terragrunt run.

[yhakbar]

Hey @ferplnat ,

Can you explain a bit further why the auth-provider-cmd doesn't meet your needs?

For context, Pipelines uses it to authenticate with AWS using OIDC, so it definitely can be used for the use-case you're trying to solve here.

[denis256]

Hi,
you can also try using the --terragrunt-no-cache flag to disable caching during execution.

For more details, see the documentation:
https://terragrunt.gruntwork.io/docs/reference/hcl/functions/#run_cmd

[ferplnat]

@yhakbar Apologies for the delay in response, I was traveling.

Hey @ferplnat ,

Can you explain a bit further why the auth-provider-cmd doesn't meet your needs?

For context, Pipelines uses it to authenticate with AWS using OIDC, so it definitely can be used for the use-case you're trying to solve here.

Sure thing, I may be coming at this wrong. The goal is to provide an internally re-usable GitHub Action, the developer consuming the GitHub action isn't supposed to worry about the auth implementation. The implementation I tried that originally didn't work with auth-provider-cmd was to use:

• TG_IAM_ASSUME_ROLE_SESSION_NAME and TG_IAM_ASSUME_ROLE_DURATION set automatically by the action, not to be tampered with.
• TG_AUTH_PROVIDER_CMD set by the action and provides only the awsRole.webIdentityToken
• iam_role programmatically defined in terragrunt.hcl (or included config, up to developer)
  • This pattern was the cleanest to support deployments across 10's of AWS accounts, each with their own role configured.

The problem is that auth-provider-cmd expects a complete configuration, which makes sense as it's supposed to run before the configuration is read to support some Terragrunt AWS functions. So it would fail on the following condition:

terragrunt/cli/commands/run/creds/providers/externalcmd/provider.go

Lines 121 to 124 in 3b29e48

	if role.RoleARN == "" {
	l.Warnf("The command %s completed successfully, but AWS role assumption contains empty required value: roleARN, nothing is being done.", opts.AuthProviderCmd)
	return nil
	}

If auth-provider-cmd was somehow updated (or already supports but I couldn't figure it out) to allow partial configurations to be later resolved after the configuration is parsed, I think that would improve flexibility with the built-in AWS role assumption logic.

Or even a TG_IAM_ASSUME_ROLE_WEB_IDENTITY_TOKEN_CMD that would allow programmatic resolution of an OIDC token.

[ferplnat]

Hi, you can also try using the --terragrunt-no-cache flag to disable caching during execution.

For more details, see the documentation: https://terragrunt.gruntwork.io/docs/reference/hcl/functions/#run_cmd

Thanks @denis256 ! This provides a workaround for the issues encountered, I appreciate the quick change. The above comment still stands if there's a more ergonomic way to approach this.