Error assuming role allow-ops-admin-access-from-other-accounts.

[zackproser]

A customer asked:

We are trying to assume a role allow-ops-admin-access-from-other-accounts using the IAM user from security account that is in the group access-all-external-account . The group has permission for various roles including allow-ops-admin-access-from-other-accounts
We executed aws-auth --role-arn "arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts" --role-duration-seconds 3600 but we are getting User: arn:aws:iam::XXXXXXXXXXXX:user/XXXXXXXX not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts
But when we assume the other role allow-auto-deploy-from-other-accounts which is also an existing permission within the group, we can assume it successfully.
Any thought on this? Do we miss something?

[zackproser]

Debug log

Confirming the IAM user, groups and permissions in question

In diagnosing the issue, we first ensured that we understood which group the IAM user that the customer was referencing was in. We confirmed it was the -access-all-external-accounts group that was defined in both vars/autogen/common_vars.yml:

... 
create_access_keys: false
create_login_profile: true 
groups: 
       - access-all-external-accounts
...

and security/_global/account-baseline/users.yml

{customer-user-name}: 
  create_access_keys: false
  create_login_profile: true 
  groups: 
  - access-all-external-accounts
  - iam-admin
  - ssh-grunt-sudo-users
  pgp_key: keybase:{customer-user-name}
...

At this point we knew that the user wanted to assumed allow-ops-admin-access-from-other-accounts which is defined in their cross_account_groups.yml and that their intended IAM user should be able to assume it (because it is a part of the access-all-external-accounts group), however the error encountered suggested the user did not have sufficient access to assume the role.

Next, we double-checked the access-all-external-accounts IAM policy, which is attached to the access-all-external-accounts group and found there all of the expected permissions to assume the various roles defined in each of the various AWS accounts.

Reproducing the users's error with aws-auth

Next, we installed aws-auth and rolled new security credentials for the intended IAM user who was attempting to assume the role. We then issued the following command on behalf of the target IAM user to attempt to reproduce the reported problem:

aws-auth --role-arn "arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts" --role-duration-seconds 3600
2021-08-19 12:24:04 [INFO] [aws-auth] Assuming role arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts. These creds will expire after 3600 seconds.)

and we were able to reproduce the reported issue:

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:iam::XXXXXXXXXXXX:user/{customer-iam-user} is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts

Confirming trust relationships and MFA settings

We also confirmed that the target account (dev) had a valid trust relationship configured with the security account. At this point we also saw the MFA setting:

Condition 	Key 	Value
Bool 	aws:MultiFactorAuthPresent 	

true 

Discovering the root cause

This means that the successful assumption of the target IAM role would require presenting a valid MFA token!

Next, we configured virtual MFA using Google Authenticator within the security account for the target IAM user and retried our same aws-auth call again, this time passing both the --serial-number from the IAM user (same as their user ID) and the --token-code flag with the One Time Password (OTP) from Google Authenticator :

aws-auth --role-arn "arn:aws:iam::XXXXXXXXXXXX:role/allow-ops-admin-access-from-other-accounts" --role-duration-seconds 3600 --serial-number arn:aws:iam::123456789011:mfa/jondoe --token-code 123456
...
Success!