In the Reference Architecture, why is there an ECS Deploy Runner in each account? Is this DRY?

[zackproser]

A customer asked if having an ecs-deploy-runner in each account is a violation of the DRY (don't repeat yourself) principle:

This is kind of an abstract, strategy question but here goes. With our refarch, we have a number of DRY violations (in particular, ecs-deploy-runner s in each account). Is it generally advisable to identify these things, then introduce a higher-level module (i.e. by wrapping the Gruntwork one) with all the commonality to reduce the size of the leaf terragrunt.hcl files?

[zackproser]

Having one ECS Deploy Runner in each account is intended

The ECS Deploy Runner (EDR or ecs-deploy-runner) actually has two purposes:

1. During the Ref Arch deployment, performed by a Gruntwork engineer, the EDR is intentionally spun up in each account. Once up, it is the responsibility of the EDR to run terragrunt run-all apply on that particular account's infrastructure-live folder (e.g., dev) in order to provision the resources defined in the scaffolded code. So, it's the EDR that is actually responsible for deploying the generated code that defined a Ref Arch
2. Once the Ref Arch is handed off, the EDR intentionally remains in each account because it is what runs the CI/CD operations (running terragrunt plan when you push a feature branch, or terragrunt apply when you merge to main) and it is a core component of what makes our product unique.

EDR is concerned with locking down CI/CD operations for enhanced security

In a traditional infrastructure CI/CD setup, you are bound to give unlimited admin privileges to the CI server so that it can deploy arbitrary infrastructure. However, this has many risks, the primary one being a user with write access can open a PR that does malicious things in the CI pipeline by editing the CI config. For example, a user could modify one of the steps in .circleci/config.yml to run the command env, which would dump all environment variables to stdout. (However it appears that CircleCI is smart enough to obfuscate credentials which are defined as Environment Variables in the Project Settings.)

To protect against this, the ecs-deploy-runner employs a limited action model, where the AWS credentials provided to the CI server is only allowed to trigger specific actions, one of them being running terraform apply on the latest infra code.

DRYing up EDR configurations

By nature, the ecs-deploy-runner is a commodity in that there isn’t much difference between all the accounts. The biggest difference is actually in the shared account, where the ecs-deploy-runner is configured to also build docker images and AMIs. However, outside of that all the other app accounts have the exact same ecs-deploy-runner configuration.

In that regard, DRY-ing up the configurations that are currently nearly identical across accounts would reduce complexity by making it less likely to forget to update all the ecs-deploy-runners.

That said, the challenge is DRY-ing up the configuration in a satisfactory way that doesn’t inadvertently introduce some other bug.

For what it's worth, we are really close to offering something better in the imports feature: gruntwork-io/terragrunt#1566