EKS cluster exhausted all our private app subnet's IP addresses

[gruntwork-support]

This message was extracted from a discussion that originally took place in Gruntwork Community Slack. Names and URLs have been removed where appropriate

From a customer

Hey Gruntwork folks! 👋

We ran into a problem in our Dev environment today where our EKS cluster exhausted all our private app subnet's IP addresses. Fun!

I was looking into this more and saw that our subnets (created with vpc-app module) are using a default prefix of /21 giving them 2048 addresses each.

Our CIDR uses /16 which effectively allows for up to 65,536 total addresses, so we have some room to grow.

My question for you is how to accomplish this. I see that I can specify private_app_subnet_cidr_blocks, but I'd rather not have to manually calculate out each CIDR block to pass in to the module for every one of our VPCs. Is there a better way? Perhaps with the subnet_spacing vars and subnet_bits vars? I tinkered a little with that but was not really sure how to calculate the proper spacing given the desired bits that I gave it.

[gruntwork-support]

From a grunt

Hello! I'm not the perfect person to answer this as I'm fairly new to calculating CIDR subnets, but I found Terraform's native functions cidrsubnet (doc) and cidrsubnets (doc) super handy.
Yep, you would use subnet_spacing and subnet_bits. To see exactly what you would get, run them in terraform console, for example:

% terraform console
> cidrsubnet("10.10.0.0/16", 4, 10)
10.10.160.0/20

Here, the subnet_bits are 4, subnet_spacing is 10. (Hashicorp calls these newbits and netnum.)
But cidrsubnets will give you a whole set of them

> cidrsubnets("10.1.0.0/16", 4, 4, 8, 4)
[
  "10.1.0.0/20",
  "10.1.16.0/20",
  "10.1.32.0/24",
  "10.1.48.0/20",
]

Yeah, super fun when you run out of IPs for the EKS cluster. What size instances are you running, out of curiosity?

[gruntwork-support]

From a customer

Very helpful thanks!

[gruntwork-support]

From a grunt

In case you're interested, there's this very arcane documentation about how many ENIs you can have per instance type. Hat tip to person who pointed this out. https://github.com/aws/amazon-vpc-cni-k8s/blob/88d17816c1e07dbdb03a546f1966574ab14356f1/misc/eni-max-pods.txt

[gruntwork-support]

From a customer

Our instances are all m5/m5a/t3/t3a.xlarge.

Thankfully we haven't hit any ENI limits yet! Our pods are a bit heavy so we end up scheduling maybe 12-15 pods max to a given node.

[gruntwork-support]

From a grunt

Yay, glad this helps!

Super long term, I have no idea what people do when they run out of IP addresses on a /16 subnet. 🤷 (with EKS in particular)

[gruntwork-support]

From a customer

Yeah, I think at that point you'd want to start dividing workloads across more VPCs/CIDRs.

[gruntwork-support]

From a grunt

EKS seems (or is) very IP hungry. If you were to exceed 65k addresses and need to scale up to multiple clusters across VPCs, what a nightmare that could be to manage. Thankfully Riskalyze (you) aren't there yet! sweat_smile But this kind of thing worries me

[gruntwork-support]

From a customer

Yeah, it definitely seems like it could do networking better and use less IPs. We could use Istio with multi-cluster mesh if we had to expand to multi-VPC, but that would certainly add some complexity to our configuration.

[gruntwork-support]

From a grunt

Jumping in a bit late...

AFAIK, there are a few alternatives to deal with the lack of IPs issue, with varying levels of difficulty:

• Add more subnets to EKS itself (I don't remember which specific configuration it is but EKS itself only use some subnets). If you create new subnets and then allocate them to be used by EKS it will use them. (Can be a band-aid solution when you still have space on your VPC and cannot migrate to a bigger one). This seems like a newer tutorial, I don't remember if we had to do all the steps, though. Maybe we used this one instead :thinking_face:
• Use an alternate compatible CNI plugin , with these you will not be restricted by the IP limitations of the AWS CNI. I've heard interesting things about these alternative CNIs like the fact that IPs and Pods do not end up being a 1:1 scenario, but as I've never used them I don't know how they work nor how complex/challenging they are to configure 😞
• I couldn't find, but I remember reading/watching some presentation about the "million pods club" where there was some debates about how to reach astounding numbers of pods in EKS

[gruntwork-support]

From a customer

Thanks person , some interesting stuff there! I think our solution is to not let our cluster get so big! Haha... 😅

[gruntwork-support]

From a grunt

You know, it is always good to be prepared when your business reach millions of customers 😉

[gruntwork-support]

From a grunt

Another solution that I recalled now is that you can play with WARM_ENI_TARGET, WARM_IP_TARGET and MINIMUM_IP_TARGET AWS CNI's variables:

• https://docs.aws.amazon.com/eks/latest/userguide/pod-networking.html
• https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/eni-and-ip-target.md

There you make an explicit tradeoff between pod' startup time and IP availability. As, if you try to start a new pod, when you have no more free IPs available on the ENI, the CNI will ask for more IPs to be attached to the ENI. Only after this is done, is that it will give the 🏁 for the pod to start. (If it falls into the ugly scenario of "no more IPs can be attached to the ENI" -> "a new ENI needs to be created and associated to the node" it can adds more 10 seconds according to AWS' docs)

This was another solution that we had to adopt to deal with the lack of IPs. On our case the added delay was completely tolerable.