Karpenter repeatedly creating launch templates, exhausting quota, during AWS outage scenario

[jhughes-mc]

Description

Observed Behavior:
AWS outage causing Karpenter to repeatedly create launch templates, exhausting account limits and preventing capacity from being created. AWS outage began on 10/20 around 1am PDT

Started on version 1.5.0, upgraded and issue persisted. Went from 1.5.0 >1.5.5> 1.8.0

│ karpenter-749b756667-dnj5q {"level":"ERROR","time":"2025-10-20T13:30:36.045Z","logger":"controller","message":"Reconciler error","commit":"31fa2ea","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeCla │
│ im","NodeClaim":{"name":"spot-gpu-tdqhw"},"namespace":"","name":"spot-gpu-tdqhw","reconcileID":"38e77c45-d816-4235-b8eb-2c2f21385517","aws-error-code":"MaxTemplateLimitExceeded","aws-operation-name":"CreateLaunchTemplate","aws-request-id":"1 │
│ 1a30a53-4686-4487-88ab-fd866640d4ef","aws-service-name":"EC2","aws-status-code":400,"error":"launching nodeclaim, creating instance, creating nodeclaim, getting launch template configs, getting launch templates, creating launch template, ope │
│ ration error EC2: CreateLaunchTemplate, https response error StatusCode: 400, RequestID: 11a30a53-4686-4487-88ab-fd866640d4ef, api error MaxTemplateLimitExceeded: Account has already reached max permitted templates (aws-error-code=MaxTemplat │
│ eLimitExceeded, aws-operation-name=CreateLaunchTemplate, aws-request-id=11a30a53-4686-4487-88ab-fd866640d4ef, aws-service-name=EC2, aws-status-code=400)"}
Expected Behavior:
Karpenter correctly recognizes problem and does not seemingly loop endlessly exhausting launch template limits.

Reproduction Steps (Please include YAML):
Not sure how, but would need to recreate the outage scenario that occured on the AWS side. I gave an example of our nodeclass/nodepool config below

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
name: default-1
labels:
app.kubernetes.io/name: default
spec:
metadataOptions:
httpPutResponseHopLimit: 2
detailedMonitoring: true
amiSelectorTerms:

alias: bottlerocket@latest
blockDeviceMappings:
deviceName: /dev/xvda
ebs:
encrypted: true
volumeSize: 20Gi
volumeType: gp3
deviceName: /dev/xvdb
ebs:
encrypted: true
volumeSize: 100Gi
volumeType: gp3
nodepool
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: default
spec:
disruption:
budgets:

• nodes: 20%
  consolidateAfter: 1m
  consolidationPolicy: WhenEmptyOrUnderutilized
  template:
  metadata:
  labels:
  lifecycle: OnDemand
  nodegroup-type: ondemand
  spec:
  expireAfter: 720h0m0s
  nodeClassRef:
  group: karpenter.k8s.aws
  kind: EC2NodeClass
  name: default-1
  requirements:
• key: kubernetes.io/os
  operator: In
  values:
• linux
• key: karpenter.k8s.aws/instance-hypervisor
  operator: In
  values:
• xen
• nitro

Versions:

• Chart Version: 1.8.0
• Kubernetes Version (kubectl version): 1.33.5

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ryan-mist]

Hey @jhughes-mc, Ill take a look at our retry mechanisms in our launch template creation paths.

If you happen to know - was the majority of the launch templates created in the account from Karpenter? Or was your account already close to launch template account limits?

[jhughes-mc]

Hey @jhughes-mc, Ill take a look at our retry mechanisms in our launch template creation paths.

If you happen to know - was the majority of the launch templates created in the account from Karpenter? Or was your account already close to launch template account limits?

We normally operate with around 150 templates in the region/account. During the AWS outage, karpenter consumed up to our limit of 5000.

[ryan-mist]

Got it. Were all the launch templates created for the same EC2NodeClass - or do you have a lot for your cluster?

Additionally, was it on v1.5.0 that the account limit was hit? Theres an additional path to create launch templates in 1.8.0 as we create 1 per NodeClass while validating.

[ryan-mist]

Can you share the logs you saw before this error occurred? I'm interested to see if Karpenter ever logged errors relating to launch template creation/deletion/describing (e.g failed to delete launch template, invalidating launch template in the cache because it no longer exists, etc.). Also, did you have to manually go in and delete these Launch Templates or did Karpenter delete them eventually?

[jhughes-mc]

Got it. Were all the launch templates created for the same EC2NodeClass - or do you have a lot for your cluster?

Additionally, was it on v1.5.0 that the account limit was hit? Theres an additional path to create launch templates in 1.8.0 as we create 1 per NodeClass while validating.

Limit was hit on v1.5.0, but even after upgrading karpenter continued to create these.

Can you share the logs you saw before this error occurred? I'm interested to see if Karpenter ever logged errors relating to launch template creation/deletion/describing (e.g failed to delete launch template, invalidating launch template in the cache because it no longer exists, etc.). Also, did you have to manually go in and delete these Launch Templates or did Karpenter delete them eventually?

Unfortunately, we only retain logs for 14 days for Karpenter. What I provided in the ticket is really all I have at this point. Karpenter did eventually delete these templates I think....we're not entirely sure because we ran a script to delete them behind Karpenter, while it was fighting us and spamming creation of launch templates. We did this because sometimes ec2 creation was successful during the outage, but these could not even be attempted when we reached quota.

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.