[Bug]: Credential is corrupted when attempting to deploy cross-regional environment

[krdchang]

Description:

I have a "production" environment deployed to ap-northeast-1. I'm trying to deploy a new "prod-tw" environment in ap-east-2. However, after initializing "prod-tw", when I try to deploy or delete the new environment I get an error indicating that my security token is invalid. Somehow I must have corrupted the token.

Details:

I've ran it multiple times with various combinations, e.g.

$ AWS_PROFILE=tokyo copilot env init --name prod-tw --profile taipei --region ap-east-2
$ AWS_PROFILE=tokyo copilot env init --name prod-tw --profile taipei
$ AWS_PROFILE=tokyo copilot env init --name prod-tw

Observed result:

$ AWS_PROFILE=tokyo copilot env deploy --name prod-tw
✘ get template version of environment prod-tw: get metadata for stack seven-prod-tw: get template summary: InvalidClientTokenId: The security token included in the request is invalid
status code: 403, request id: d8f34020-03cf-4348-9656-1e10ef918b25
$

$ AWS_PROFILE=tokyo copilot env delete --name prod-tw
Sure? Yes
✘ find service cloudformation stacks: UnrecognizedClientException: The security token included in the request is invalid
status code: 400, request id: 9a982d3f-119e-4a08-a461-4d8379a77bd5
$

I have to manually cleanup the stacks. E.g.

1. Empty the S3 artifact bucket
2. Access StackSet in Tokyo to delete the associated stack in ap-east-2.

Expected result:

Should deploy new environment successfully, e.g. here is a sample deployment of "prod-kr" (to Korea)
$ AWS_PROFILE=tokyo copilot env deploy --name prod-kr
✔ Proposing infrastructure changes for the seven-prod-kr environment.

• Creating the infrastructure for the seven-prod-kr environment. [update complete] [68.3s]
  • An ECS cluster to group your services [create complete] [6.2s]
  • A security group to allow your containers to talk to each other [create complete] [2.0s]
  • An Internet Gateway to connect to the public internet [create complete] [18.6s]
  • A resource policy to allow AWS services to create log streams for your workloads. [create complete] [4.5s]
  • Private subnet 1 for resources with no internet access [create complete] [2.0s]
  • Private subnet 2 for resources with no internet access [create complete] [4.3s]
  • A custom route table that directs network traffic for the public subnets [create complete] [11.9s]
  • Public subnet 1 for resources that can access the internet [create complete] [4.3s]
  • Public subnet 2 for resources that can access the internet [create complete] [4.3s]
  • A private DNS namespace for discovering services within the environment [create complete] [43.5s]
  • A Virtual Private Cloud to control networking of your AWS resources [create complete] [14.3s]

Debugging:

I'm using a permanent accessKey and secretAccessKey for my IAM user account, i.e. does not expire (and also has AdminAccess privileges). When executing,
$ AWS_PROFILE=tokyo copilot env init --name prod-tw --profile taipei
The taipei profile has the same accessKey and secretAccessKey as the tokyo profile (the only difference between profiles is the region).
Using the taipei profile, e.g.
$ AWS_PROFILE=taipei copilot env deploy --name prod-tw
does not work because copilot expects to find /copilot/applications/seven/environment/prod-tw in ap-northeast-1 (i.e. the region for which the StackSet seven-infrastructure resides).
It seems to me that a credential is being stored someplace and doesn't have the permission to empty the S3 artifact bucket, hence why deleting the environment fails (and the same reason why deployment fails as well).

[krdchang]

This issue reads similar to #4377

[krdchang]

I'm using copilot-cli version 1.33.3

[krdchang]

From a different AWS account, enabling the same region (ap-east-2), I'm getting the same issue when attempting to run,
$ AWS_PROFILE=tokyo copilot env deploy --name prod-tw

Therefore, it appears to be an issue with the new enabled region (ap-east-2 in my case).