[Feature]: documentation possibly needed for AWS IAM user race condition

[andrewpatto]

Terraform Version

1.12.2

Provider Version

0.25.2

Seqera Platform Type

Seqera Enterprise v24.2

Affected Resource(s)

seqera_compute_env

Expected Behavior

This is not actually a bug with the Seqera terraform provider - but might be enough of a common issue that possibly just needs some documentation.

If I create an IAM user and Seqera credentials and compute environment in a single Terraform stack - I expect that they will all succeed.

Actual Behavior

If an AWS IAM user is created at the same time (i.e. in the same terraform) as the Seqera credentials and compute env - the "setting up" of the IAM user in the AWS internal systems will not have completed immediately.

So when those credentials are straight away used by Batch Forge for setting up the compute environment - it fails with a variety of 403 errors at different points.

Terraform Configuration

resource "time_sleep" "wait_for_user_complete" {
  create_duration = "30s"
  depends_on = [
    aws_iam_user_policy_attachment.attach_policies_to_user_1,
    aws_iam_user_policy_attachment.attach_policies_to_user_2,
    aws_iam_access_key.batch_forge_access_key
  ]
}

resource "seqera_credential" "credential" {
  depends_on = [
    time_sleep.wait_for_user_complete
  ]

  name          = "seqera-credentials-${var.short_name}"
  description   = "AWS account credentials for compute environment"
  provider_type = "aws"
  workspace_id  = seqera_workspace.workspace.id

  keys = {
    aws = {
      discriminator = "aws"
      access_key    = aws_iam_access_key.batch_forge_access_key.id
      secret_key    = aws_iam_access_key.batch_forge_access_key.secret
    }
  }
}

Terraform Output/Error

N/A

Debug Logs (Optional)

Steps to Reproduce

I have attached the needed Terraform (an interim "wait" resource) that solves the issue. Again, not saying this is actually a bug with the seqera terraform provider - but was non-obvious as to why it was failing.

Maybe this could go in a "common issues" or FAQ or something.

Additional Context

No response

[andrewpatto]

And alternative might be some form of retry/back-off semantics in the Compute Environment resource provider..

[andrewpatto]

by IAM user create I mean

resource "aws_iam_user" "batch_forge_user" {
  name = "seqera-${var.short_name}-batch-forge-user-"
}

resource "aws_iam_access_key" "batch_forge_access_key" {
  user = aws_iam_user.batch_forge_user.name
}

... standard policies ...

[gavinelder]

Hi @andrewpatto thanks for this feedback over a number of issues.

For this specific case I have raised an internal FR to our platform core team for server side retries on compute recreate to see if that is possible with a short backoff.

For the resource employing a depends on is not always viable given AWS API calls can be a-sync and it's probably best to leverage a time_sleep resource

We will update our examples in the documentation.

[gavinelder]

We are currently investigating back-off and retry on compute environment creation for specific error types and messages to help improve this user experience.