Certificates pending when trying to use parent and child route53 modules

[lorelei-rupp-imprivata]

Wondering if gruntworks has seen this before, or has some ideas around best practices. We use the terraform-aws-service-catalog/networking/route53 module to first create "parent route53 resources" and then we use terragrunt, with a dependency to create child route53 resources using those parents. Our pipelines apply the parent module fine. However when it goes to apply the child. it ALWAYS fails with below. We think we need some sort of wait or something inbetween these but haven't been able to pin point where it goes or "how long". I have tried a null resource to run after the parent is created, but that doesn't seem to do the trick. Also once it fails, if I apply again on the child route53 module it always passes

module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["rds.XXX"]: Still creating... [44m50s elapsed]
module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["clb.XXX"]: Still creating... [44m50s elapsed]
module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["clb.XXX"]: Still creating... [44m50s elapsed]
module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["sys.XXX"]: Still creating... [45m0s elapsed]
╷
│ Error: Error describing created certificate: Expected certificate to be issued but was in state PENDING_VALIDATION
│ 
│   with module.route_53_zone_creation[0].module.acm-tls-certificates.aws_acm_certificate_validation.cert["clb.XXX"],
│   on .terraform/modules/route_53_zone_creation.acm-tls-certificates/modules/acm-tls-certificate/main.tf line 125, in resource "aws_acm_certificate_validation" "cert":
│  125: resource "aws_acm_certificate_validation" "cert" {
│ 

---

Tracked in ticket #109114

[zackproser]

Hi Lorelei,

When I've encountered this issue in the past, it was always due to some subtle mis-configuration in my code or my actual Route 53 domains, nameservers and Route53 Public Hosted Zones.

I'll link the official AWS doc on how DNS-based certificate validation works which is worth reading if you haven't already.

In addition, here's a couple of the things that I've needed to ensure in the past to make sure programmatic DNS validation worked properly:

1. The domain you're issuing ACM certificate requests against must be a valid, registered domain that you control.
2. If the domain name you're trying to generate a certificate for was registered via Route53, ensure that the name servers of the domain itself match the name server records in the Route53 hosted zone that gets created to match it. In other words, it's possible, and a common source of this error, to register example.com via Route53, which will have a set of NS records randomly assigned to it, and then create a public Route 53 Hosted Zone named example.com which then has a different set of NS records randomly assigned to it. If you do not reconcile these NS records, public resolution will fail, and so programmatic validation via DNS will fail within AWS. The NS records set on the domain entry itself in Route 53 must match those in the Public Hosted Zone associated with the domain.
3. You can test the programmatic DNS validation process "out of band" by running dig queries against the CNAME validation challenge records that get inserted in your Route53 Public Hosted Zone. You should see records similar to random_value>.acm-validations.aws being inserted by our Terraform module. Importantly, _these records must publicly resolve over the internet_as that is how AWS tests them during validation. If the dig queries fail from your machine, they're going to fail during the validation process, and you'll get stuck at this validation pending step until the timeouts are reached. You can also use a tool such as https://www.whatsmydns.net/, setting your record type to CNAME and entering the validation challenge record directly in the search bar. When a few resolvers are able to respond with your records - this is a sign that validation will likely eventually succeed. Conversely, if no resolvers return your record after a few minutes, it's likely you have a Route53 misconfiguration issue preventing your domain (and its newly inserted records) from resolving correctly over the public internet.
4. Mis-configured requests for wilcard certificate requests might not match the common name of the domain - be careful if you're requesting wildcard domains in your tests, for example.

Hope this helps and gives you something more to dig into!

[lorelei-rupp-imprivata]

Thanks, I may try some of these things when I am in this state. What is odd about this, is when this happens, all I have to do is re apply the module and it works the second time. Without any changes on my behalf. So I don't think the NS can be misconfigured because I don't actually change anything. Unless the first apply didn't do something correctly perhaps and the second apply does?

[zackproser]

Hi Lorelei,

Understood - thank you for that clarification. It does make me recall a Terraform bug that sounds similar where certain resources require being applied twice. I'm looking for it now, and if I can find it I'll link to it here.

I wonder:

1. Are you able to view the Route53 console and your Public Hosted Zones while your first apply is running and confirm that valid-looking validation records are being created on the first apply? or do they only appear after the second apply?
2. This is not really satisfactory, but in the interim perhaps the workaround is to run apply twice for this workflow.

[lorelei-rupp-imprivata]

Yeah actually I was just doing that and I think I know what is going on. It looks to be an issue on our side, because of a terragrunt limitation that we have a workaround in place, which the side effect of the workaround was this.

So that "child" route 53 terragrunt, needs to use "locals" to do some logic to create the public zones to pass to the GW module. Therefore we need access to the parents outputs in the locals. We were following workarounds described here gruntwork-io/terragrunt#1487 and I believe this is how we get into this situation.
While it looks like we are correctly passing the parents hosted zone IDs in, they are actually set to null.

This is why the second apply works.

I am trying to see if there is another way to solve this issue with terragrunt + locals needing dependency output

[lorelei-rupp-imprivata]

Thanks, I have solved the issue, without your debug recommendations I never would have gone down that path, so hopefully this helps someone else in the future

[zackproser]

Hi @lorelei-rupp-imprivata Awesome - glad to hear you got this sorted. Thanks for closing the loop!