Using "iam_role" with S3 state bucket in another account

[tcourtnage-tbl]

Hi. I'm having a heck of a time with something that I swear worked fine two days ago. Now I can't explain the failure.

I'm trying to use the iam_role feature to have Terragrunt deploy resources into another AWS account and keep my remote state DRY.

My terragrunt.hcl at the root of my project. my-state-bucket is in account 111111111111.

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket = "my-state-bucket"

    key = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

My terragrunt.hcl in a subfolder:

include "parent" {
  path = find_in_parent_folders()
}

iam_role = "arn:aws:iam::222222222222:role/my-cross-account-role"

terraform {
  source = "tfr://registry.terraform.io/........"
}

inputs = {
....
}

my-cross-account-role has AdminstratorAccess, and has a policy that allows s3:* access to my-state-bucket and kms:* to the CMK KMS. my-state-bucket also has a bucket policy that allows s3:* for the arn:aws:iam::222222222222:role/my-cross-account-role principal. The KMS key also has a policy that allows kms:* to the principal as well.

When I run terragrunt plan --terragrunt-log-level debug, I get this:

...
DEBU[0000] Assuming IAM role arn:aws:iam::222222222222:role/my-cross-account-role with a session duration of 0 seconds.
....
DEBU[0000] Initializing remote state for the s3 backend  prefix=[.....] 
DEBU[0000] Checking if SSE is enabled for AWS S3 bucket my-state-bucket  prefix=[.....] 
DEBU[0001] Checking if bucket my-state-bucket is have root access  prefix=[.....] 
DEBU[0001] Could not get policy for bucket my-state-bucket  prefix=[.....] 
DEBU[0001] Checking if bucket my-state-bucket is enforced with TLS  prefix=[.....] 
ERRO[0001] MethodNotAllowed: The specified method is not allowed against this resource.
	status code: 405, request id: EH84T2FY841KD420, host id: YSYFjAWjZ1mYp7C4L5h1v7VV6A38ff6mdK9MY05o1h424dGlUrl21Cif5vGugZlIl/OFhmQkCZY= 
ERRO[0001] Unable to determine underlying exit code, so Terragrunt will exit with error code 1 

I can assume the my-cross-account-role at the cli, and can successsfully access the bucket, list objects, put and get objects and everything is good. However, if I try this, it fails, similar to the error above:

$ aws s3api get-bucket-policy --bucket my-state-bucket

An error occurred (MethodNotAllowed) when calling the GetBucketPolicy operation: The specified method is not allowed against this resource.

The only thing I can come up with is https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketPolicy.html, a 405 Method Not Allowed because the identity doesn't belong to the bucket owner's account. As mentioned, I'm 99.9999% sure this was working two days ago, and after many changes and things moving around, now I can't figure out what's going on!

Happy to provide more details if I'm missing some details above.

---

Tracked in ticket #109144

[yorinasub17]

Did you recently update terragrunt to the latest version? This may be an issue with the introduction of the bucket state syncing feature in Terragrunt.

Can you try setting disable_bucket_update = true on the remote_state block and see if that avoids the issue? E.g.:

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket = "my-state-bucket"

    key = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
  disable_bucket_update = true
}

[tcourtnage-tbl]

Thanks @yorinasub17! That did it. To answer your question, however, I only just started using Terragrunt in the past week, so I have been on the latest version (v0.38.7) the whole time (no terragrunt update involved).

[yorinasub17]

Oh hmm that would be odd... Did you by chance update the way Terragrunt is used?

In any case, if that did indeed solve the issue, then the issue lies in an update to the way Terragrunt operates since version v0.37.0. Starting with v0.37.0, Terragrunt attempts to sync the bucket with the configuration parameters used in Terragrunt (e.g., setting up SSE). Because of this, Terragrunt requires access to the bucket metadata to check and compare if the bucket matches what Terragrunt expects.

Unfortunately, with cross account bucket access, you can't check or adjust those metadata parameters across accounts, which causes Terragrunt to fail. Hence, disabling this routine fixes the issue (since it isn't compatible with your use case).

Hope that gives some more context as to what is going on, and perhaps give a lead on what might have caused the problem in your setup!

[tcourtnage-tbl]

Oh hmm that would be odd... Did you by chance update the way Terragrunt is used?

Nope, I haven't done anything there.

The S3 bucket was not created by Terragrunt, it was manually created (with Terraform) before I started using Terragrunt. However, your explanation makes some sense.

Appreciate the quick response!

[gzjon]

disable_bucket_update inside the remote state config block (terragrunt version v0.39.1):

remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket = "my-state-bucket"

    key = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
    disable_bucket_update = true
  }
}

[OlesYudin]

@tcourtnage-tbl Hello, do you have any issues with dynamoDB cross account or you create new dynamoDB table for each account?

[tcourtnage-tbl]

@OlesYudin I'm creating a new DynamoDB for each account. That seems to be the easiest way to make it work.

[OlesYudin]

@tcourtnage-tbl I know that it is a best solution. But, have you ever try to configure only one DynamoDB table and try to have access to it cross account.
I can locally run terragrunt in account_2 with dynamoDB in account_1. But when I try to run it on pipeline I`ve got error. Terragrunt try to find dynamoDB on account_1. Also I can say that in pipeline I auth to account_1 and account_2. Maybe you know how to prevent this behaviour?

[OlesYudin]

So, finally I understand reason of my error. Add description here.