In the RefArch Gruntwork pipelines.yml, why are plan and deploy bundled in the same job?

[umm0n]

r:terraform-aws-ci-pipeline-example

The Gruntworks pipe for the ref arch has plan and deploy steps bundled in the same job. This causes plan to be run when merging against master, even if it was ran moments ago when pushing the changes against the branch. For large plans, this creates a big overhead in terms of time. For example:

1. Create PR, push changes. Pipeline runs plan, it takes 30min.
2. Merge PR. Pipeline runs plan, it takes 30min. Then pipe runs deploy, takes another 30mins.

Why are we running plan again when merging? We already know the plan, it ran as a result of creating the PR in the first place. Wouldn't it make more sense to have these steps split into separate jobs? For longer plans, the time savings could be huge when merging.

---

Tracked in ticket #109794

[josh-padnick]

One of our engineers responded internally with the following:

Terraform will run plan under the hood whenever you run apply - unless you give it the path to a saved plan output. It needs to do this in order to have the latest plan before applying. To fully answer this question, we need confirm that this is the same behavior that occurs with the ECS Deploy Runner's (Gruntwork Pipelines') internal usage for deploying new Reference Architectures.

I'll also add that you're encouraged to re-configure the pipelines configurations however you'd like. If you want to pass a plan output between steps to avoid running plan again, you could use something like CircleCI Artifacts or GitHub Actions Artifacts.

Once we've completed the internal research, we'll update this thread again.

[umm0n]

Thanks Josh. I'm actually asking this because we have indeed created a PR to re-configure the pipeline and remove plan when merging, but we wanted to be aware if there was any reason in particular why this was configured so.

[umm0n]

The images below might make it more clear. These are the pipelines that ran when merging a particularly big PR we had a couple weeks ago. First, the last commit we pushed against the branch triggered the pipeline, which ran only plan:

And when we merged the branch, the pipeline ran plan and apply (deploy):

If the pipe had ran only apply (deploy) on merge, it would have saved us around 20mins.

[ellisonc]

The intent for re-running "plan" before deploy is to provide an updated status to review before approving the deployment. This can be useful if changes occurred on the main branch after the initial PR was created. In environments where there is no approval step prior to the deploy (dev, stage, ...) re-running plan is likely unnecessary. As Josh described in his answer, we encourage you to modify your pipeline so it works best for your needs!

[josh-padnick]

Thanks for those very helpful screenshots. Taking 20 mins to do a terraform plan is unusual. Can you share any details on what kind of operations you're doing in that Terraform code? That won't change our responses to this question, but understanding the context is helpful.

[umm0n]

Sure! I'll start by clarifying that most of our plans take anywhere between 2 to 10 mins, these longer plans are more unusual.
In this particular case, the PR involved changes to two services in envcommon, which resulted in terraform having to run plan in 6 directories (3 aws envs x 2 services). Changes to envcommon files is more often than not the main cause of long plans, for us.

[josh-padnick]

Ah, interesting. We are actually about to release a beta version of a tool to streamline IaC updates (e.g. from CIS v1.4 to CIS v1.5). We're starting off by targeting specific, important updates (up next is helping customers adopt Launch Templates to replace Launch Configs), but eventually we'd like most updates to go through the tool under the premise that it makes life easier. I'll share your message with one of our product managers to make sure he's aware of this pain point -- that envcommon updates can be especially cumbersome.