Parsing yaml to get one account ID for a specific use-case is difficult

[nitrocode]

Problem

For example, segmentio has 2 sources and 2 account ids

known_aws_accounts/accounts.yaml

Lines 12 to 14 in 26925a7

	- name: 'SegmentIO'
	source: ['https://segment.com/docs/destinations/amazon-s3/', 'https://segment.com/docs/destinations/amazon-kinesis/']
	accounts: ['107630771604', '595280932656']

I assume each account id comes from its respective source.

If that's the case, I'd have to figure out which source relates to my use-case, and then get the respective account id. This is a little tricky the way the yaml is structured.

Proposal

What do you folks think about this format instead?

- name: 'SegmentIO'
  purpose:
    s3:
      source: 'https://segment.com/docs/destinations/amazon-s3/'
      account: '107630771604'
    kinesis:
      source: 'https://segment.com/docs/destinations/amazon-kinesis/'
      account: '595280932656'

or even with a full map, no list. This has an additional benefit of having its vendor names alphabetized.

accounts:
  segmentio:
    name: 'SegmentIO'
    purpose:
      s3:
        source: 'https://segment.com/docs/destinations/amazon-s3/'
        account: '107630771604'
      kinesis:
        source: 'https://segment.com/docs/destinations/amazon-kinesis/'
        account: '595280932656'

Now I can do something like this

>>> external_accounts["segmentio"]["purpose"]["s3"]["account"]
'107630771604'

Notes

• I noticed is that source is sometimes a static URL and other times it's an array. The above proposal can fix this issue too.
  

  known_aws_accounts/accounts.yaml

  Lines 9 to 14 in 26925a7

  	- name: 'Cloudhealth'
  	source: 'https://github.com/mozilla/security/blob/master/operations/cloudformation-templates/cloudhealth_iam_role.json'
  	accounts: ['454464851268']
  	- name: 'SegmentIO'
  	source: ['https://segment.com/docs/destinations/amazon-s3/', 'https://segment.com/docs/destinations/amazon-kinesis/']
  	accounts: ['107630771604', '595280932656']

• Sometimes a source is missing
  

  known_aws_accounts/accounts.yaml

  Lines 228 to 230 in 26925a7

  	- name: 'Rockset'
  	accounts: ['216690786812']
  	- name: 'CloudHiro'

• Sometimes more than one account ID is in a vendor with ONLY a single source which is OK in the above proposal too
  

  known_aws_accounts/accounts.yaml

  Lines 172 to 174 in 26925a7

  	- name: 'Qualys Cloud View'
  	source: 'https://qualysguard.qualys.com/qwebhelp/fo_portal/scans/ec2_connector.htm'
  	accounts: ['080595016317', '205767712438']

• There are deprecations in titles
  

  known_aws_accounts/accounts.yaml

  Lines 326 to 328 in 26925a7

  	- name: '[Deprecated] AWS Log delivery Service'
  	source: 'https://forums.aws.amazon.com/thread.jspa?messageID=629256'
  	accounts: ['858827067514']

  • These can instead be in the yaml. Preferably not a deprecated: true because that name would be set in the "negative" instead of "positive"

    # simple enabled/disabled, if disabled, assume deprecated
    enabled: false
    # if it's current or not, if not, assume deprecated
    current: false
    # if this key is set, assume deprecated
    deprecated_date: 2025-12-31
    # or maybe a last updated_date and then all deprecated accounts can be moved
    # to a separate file such as accounts_deprecated.yaml
    updated_date: 2024-12-31

[nitrocode]

cc: @ramimac thanks for merging my PRs!

I was wondering if you could share your thoughts here if you have time

[new23d]

There could be multiple sources for the same account IDs as well. So, just from a best practices point of view, might as well keep the field type to list/array.

Also, not sure 'purpose' is clearly defined or only one at a time.

Agree with your deprecation schema though. Just my 2c.

[nitrocode]

There could be multiple sources for the same account IDs as well. So, just from a best practices point of view, might as well keep the field type to list/array.

My thinking was that I needed to get a single aws account for a particular purpose. I would find it strange if there are multiple aws accounts for the same purpose.

Also, not sure 'purpose' is clearly defined or only one at a time.

Maybe this isn't a good key. I was thinking if there are multiple aws accounts, there must be multiple uses or purposes for each account. In the example I showed above I illustrated that segmentio has two different accounts, one for s3 usage, and one for kinesis usage.

Agree with your deprecation schema though. Just my 2c.

Great!

Very good discussion. Thank you for your comments and feedback.

[christophetd]

purpose definitely sounds too specific and hard to maintain.

[ramimac]

Hey, frankly, I feel like the lack of engagement here might be due to this issue being a bit overloaded.

Responding to some of the specifics

If that's the case, I'd have to figure out which source relates to my use-case, and then get the respective account id. This is a little tricky the way the yaml is structured.

Could you clarify the user story here? I think we mostly expect folks to be doing lookups based on Account ID, or occasionally looking by vendor manually. I'm not as clear the case where you'd want to start from a source and link it to a specific account ID

For related reasons, we don't version docs currently, and have chosen to retain account IDs that may not be publicized (or used!) any more.

I noticed is that source is sometimes a static URL and other times it's an array. The above proposal can fix this issue too.

A PR fixing consistency would be very welcome here! It doesn't actually couple with a format change though

Sometimes a source is missing

AFAIK these are all from legacy account IDs. We've err'd on the side of retaining the useful context, even though it doesn't meet our current standards for inclusion

There are deprecations in titles

This sounds like a nice adjustment. Similar to above, it doesn't require a complete format revamp to incrementally improve.

Responding to the format proposal

My biggest concern would be avoiding a breaking change unless there is outsized value presented. I don't currently, personally feel like this reaches that bar. I totally agree that there would be benefits and improvements with your suggested format, and we would have been well served by nailing it better from the get!

[nitrocode]

purpose definitely sounds too specific and hard to maintain.

The data already contains a list of sources and a list of account ids. How would it be difficult to maintain?

How would you distinguish account ids?

---

Hey, frankly, I feel like the lack of engagement here might be due to this issue being a bit overloaded.

Apologies, I was trying to make a clear case for the PR title. I may have added too much information. Let me restart with a problem statement.

Context

You folks have a yaml format that seems to be based on previous formats and this is the most well maintained data of public account ids.

The notes are just call outs of what I've seen when digging into the data. My goal is to remove hard coded account IDs from our code by migrating those to a programmatic usage without having to maintain a separate data source. For example, we need a way to get a single account id for a vendor in an S3 bucket policy or in a KMS key policy. It's challenging if there are multiple account IDs per external account without clarity such as use-case or purpose to describe it.

The existing list of maps is easy for humans to search through, however, it has some challenges for programmatic access. In fact, some of the call outs above under notes, I was able to call out because I had started playing with the existing data in python, converted the data to the format above, and noticed various inconsistencies and raised a few PRs.

Technically, yes, we can do something like this, but it is cumbersome because we'd have to have knowledge of the index of the account ID of an external account that we'd want to use.

>>> [account for account in external_accounts if account["name"] = "SegmentIO"].accounts[0]
'107630771604'
>>> [account for account in external_accounts if account["name"] = "SegmentIO"].accounts[1]
'595280932656'

Here is a better example, for segmentio, the doccs show this policy containing 107630771604

{
	"Version": "2008-10-17",
	"Id": "Policy1425281770533",
	"Statement": [
		{
			"Sid": "AllowSegmentUser",
			"Effect": "Allow",
			"Principal": {
				"AWS": "arn:aws:iam::107630771604:user/s3-copy"
			},
			"Action": "s3:PutObject",
			"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/segment-logs/*"
		}
	]
}

With the current implementation of list(map), we'd need to do something like this (pseudo code, reduced spacing for clarity)

- "AWS": "arn:aws:iam::107630771604:user/s3-copy"
+ "AWS": "arn:aws:iam::${[account for account in external_accounts if account["name"] = "SegmentIO"].accounts[0]}:user/s3-copy"

with a map(map) and individual use-case account-ids, we can do something like this (pseudo code, reduced spacing for clarity)

- "AWS": "arn:aws:iam::107630771604:user/s3-copy"
+ "AWS": "arn:aws:iam::${external_accounts["segmentio"]["purpose"]["s3"]["account"]}:user/s3-copy"

---

Note that we don't have to change the existing format, if people want to continue using it. Instead, we can have a new format, like the one above or similar, and use a pipeline that outputs the current format, thus moving the source of truth for maintainability.

Proposal

A new format that resolves the main issue: distinguishing account IDs for a use-case. The new format doesn't have to the be one I proposed above.

1. Create accounts_v2.yaml with a new format that distinguishes account ids
2. Create github actions
   • .github/workflows/build.yaml that runs a custom shell script to read accounts_v2.yaml to output accounts.yaml

Would this be palatable?

---

Thanks for the comments!

[ramimac]

The data already contains a list of sources and a list of account ids. How would it be difficult to maintain?

There are a few concerns that come to mind with defining "purpose." The first is that it's not a one-to-one relationship. Vendors can have one or more account IDs, tied to one or more services/features. In some cases, vendors don't actually distinguish publicly what that mapping might be.

It also adds another temporal dimension, which is something we've avoided. For example, that might imply a process to check the documentation and update purpose if new use cases are added.

Finally, purpose would be a very subjective and open to yak-shaving. Any new field increases the bar for contributions. You'd still need a human to find the right purpose to use in code, it's not self-documenting.

How would you distinguish account ids?

I think mapping account IDs to the originating source is a reasonable middle group, ideally with timestamping and an archiving process to make sure documentation is reproducible.

---

Thanks for the time invested in rephrasing and sharing more, this was helpful in building up context on how folks are thinking about this repo!

My goal is to remove hard coded account IDs from our code by migrating those to a programmatic usage without having to maintain a separate data source. For example, we need a way to get a single account id for a vendor in an S3 bucket policy or in a KMS key policy

On a personal basis, I'm pretty skeptical of the benefit here. In my previous roles, we've used an internal mapping of AccountID to vendor/purpose/etc., to allow more human-readable policies and to improve SAST.

But directly using known_aws_accounts outsources a fair amount of trust, and it's not clear to me the benefit vs maintaining data internally containing the subset you are actively using?

The existing list of maps is easy for humans to search through, however, it has some challenges for programmatic access

Agreed!

Fundamentally, I'm skeptical that long term we'll be able to trade off supporting both human and programmatic legibility in the raw data set.

There was a great step towards offering a static site for human usage in #9, which is something we could pick back up.

My perspective is that we should optimize the underlying data for programmatic adoption - as you've been helping us evaluate!

With the current implementation of list(map), we'd need to do something like this (pseudo code, reduced spacing for clarity)

Just for emphasis, and I know this is pseudocode - we provide no ordering consistency guarantees on the data right now, making this unsuitable for another reason.

---

I've opened a few issues to track more concrete tasks that are alluded to above
#48
#49
#50
#51