NodeClaim resources for EC2 instances are deleted when instance is ShuttingDown, not Terminated

[brandond]

Description

Observed Behavior:

When deleting NodeClaims, the instance deletion is marked as complete once the instance state transitions to "Shutting Down".

Ref:

karpenter-provider-aws/pkg/providers/instance/instance.go

Lines 235 to 253 in 1ad0d78

	func (p *DefaultProvider) Delete(ctx context.Context, id string) error {
	out, err := p.Get(ctx, id, SkipCache)
	if err != nil {
	return err
	}
	// Check if the instance is already shutting-down to reduce the number of terminate-instance calls we make thereby
	// reducing our overall QPS. Due to EC2's eventual consistency model, the result of the terminate-instance or
	// describe-instance call may return a not found error even when the instance is not terminated -
	// https://docs.aws.amazon.com/ec2/latest/devguide/eventual-consistency.html. In this case, the instance will get
	// picked up by the garbage collection controller and will be cleaned up eventually.
	if out.State != ec2types.InstanceStateNameShuttingDown {
	if _, err := p.ec2Batcher.TerminateInstances(ctx, &ec2.TerminateInstancesInput{
	InstanceIds: []string{id},
	}); err != nil {
	return err
	}
	}
	return nil
	}

As per the Karpenter docs at https://karpenter.sh/docs/concepts/disruption/#termination-controller, Kubernetes Node deletion is blocked by a finalizer until the NodeClaim is gone:

Begin evicting the pods on the node with the Kubernetes Eviction API to respect PDBs, while ignoring all static pods, pods tolerating the karpenter.sh/disrupted:NoSchedule taint, and succeeded/failed pods. Wait for the node to be fully drained before proceeding to Step (3).
While waiting, if the underlying NodeClaim for the node no longer exists, remove the finalizer to allow the APIServer to delete the node, completing termination.
Terminate the NodeClaim in the Cloud Provider.
Remove the finalizer from the node to allow the APIServer to delete the node, completing termination.

As karpenter-provider-aws considers the instance to be deleted as soon as it transitions to "Shutting Down", the NodeClaim resource will be deleted, triggering removal of the finalizer from the Node resource, so the Node deletion may succeed while Kubernetes components are still running on the host.

K3s does not currently handle this well - when its Node resource is deleted, it exits, and then is restarted by systemd. When it restarts it re-registers with the cluster, and creates a new Node resource. This causes the Node (and other supporting resources) to thrash, until the EC2 instance ACTUALLY finishes shutting down and terminates.

This was reported by a K3s user who is using Karpenter to manage nodes on EC2, and is tracked on the K3s side at k3s-io/k3s#13072.

Note that we consider the behavior of the restarting the K3s Kubelet and re-registering a Node when the Node resource is deleted to be working as designed - if the user wants K3s to no longer run on the host, they should stop the service or terminate the instance.

Expected Behavior:

The instance state should actually be terminated before NodeClaim deletion is successful.

Reproduction Steps (Please include YAML):

n/a, issue was reported by user.

Versions:

• Chart Version:
• Kubernetes Version (kubectl version):

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jmdeal]

Do you know what version of Karpenter this was? Current versions of Karpenter don't just wait for the instance to be in a shutting-down state, we also wait for ec2:DescribeInstances to not return the instance with the following filters:

karpenter-provider-aws/pkg/providers/instance/instance.go

Lines 67 to 76 in da4f938

	instanceStateFilter = ec2types.Filter{
	Name: aws.String("instance-state-name"),
	Values: []string{
	string(ec2types.InstanceStateNamePending),
	string(ec2types.InstanceStateNameRunning),
	string(ec2types.InstanceStateNameStopping),
	string(ec2types.InstanceStateNameStopped),
	string(ec2types.InstanceStateNameShuttingDown),
	},
	}

This check is made here in the termination controller:

https://github.com/kubernetes-sigs/karpenter/blob/main/pkg/controllers/node/termination/controller.go#L282-L285

The result of this is that Karpenter waits for the instance to enter the Terminated state, not ShuttingDown.

[brandond]

Thanks for that pointer, I'll see if I can find out!

[Tejeev]

Our customer may be seeing problems with both 1.3.4 & 1.4.1, though that said, I may not correctly understand the impact and conflating with k3s-io/k3s#13072.
What version was the fix introduced?

[jmdeal]

This has been Karpenter's behavior since v1.0, on previous versions we deleted the node as soon as we initiated termination.

[brandond]

@Tejeev if we could get logs from both Karpenter, and from journald on a node that has been terminated by Karpenter, that might help nail down the sequence of events. Note that grabbing logs from the terminated node would probably require them to have some sort of log shipping in place beforehand.