kubergrunt eks │ sync-core-components failing during eks k8s upgrade from 1.21 to 1.22.

[sewmiuraj]

Hello,

I am currently upgrading our eks cluster (managed nodes) from 1.21 to 1.22. During upgrading the control plane - terragrunt apply was completed with errors. My cluster shows it was upgraded to 1.22, however some changes related to sync-core-components were not successfully applied.

My plan looks as follows:

# module.eks_cluster.null_resource.customize_aws_vpc_cni[0] will be created
  + resource "null_resource" "customize_aws_vpc_cni" {
      + id       = (known after apply)
      + triggers = {
          + "eks_cluster_endpoint"           = "https://XXX.eu-central-1.eks.amazonaws.com"
          + "enable_prefix_delegation"       = "true"
          + "minimum_ip_target"              = null
          + "sync_core_components_action_id" = (known after apply)
          + "warm_ip_target"                 = null
        }
    }

  # module.eks_cluster.null_resource.sync_core_components[0] is tainted, so must be replaced
-/+ resource "null_resource" "sync_core_components" {
      ~ id       = "XXXX" -> (known after apply)
        # (1 unchanged attribute hidden)
    }

  # module.eks_cluster.null_resource.wait_for_api (deposed object 722484ed) will be destroyed
  # (left over from a partially-failed replacement of this instance)
  - resource "null_resource" "wait_for_api" {
      - id       = "XXXX" -> null
      - triggers = {
          - "eks_cluster_endpoint" = "https://XXX.yl4.eu-central-1.eks.amazonaws.com"
          - "endpoint_access"      = "true"
          - "k8s_version"          = "1.21"
        } -> null
    }

Plan: 2 to add, 0 to change, 2 to destroy.

When I apply I am receiving the following error:

 Error: local-exec provisioner error
│ 
│   with module.eks_cluster.null_resource.sync_core_components[0],
│   on .terraform/modules/eks_cluster/modules/eks-cluster-control-plane/main.tf line 145, in resource "null_resource" "sync_core_components":
│  145:   provisioner "local-exec" {
│ 
│ Error running command '/opt/homebrew/bin/kubergrunt eks
│ sync-core-components --eks-cluster-arn
│ arn:aws:eks:eu-central-1:xxxx:cluster/xxx --wait   ':
│ exit status 1. Output: [] time="2023-01-09T19:27:53+01:00" level=info
│ msg="Looking up deployed Kubernetes version" name=kubergrunt
│ [] time="2023-01-09T19:27:53+01:00" level=info msg="Retrieving details for
│ EKS cluster arn:aws:eks:eu-central-1:xxxx:cluster/xxxx"
│ name=kubergrunt
│ [] time="2023-01-09T19:27:53+01:00" level=info msg="Detected cluster
│ deployed in region eu-central-1" name=kubergrunt
│ [] time="2023-01-09T19:27:53+01:00" level=info msg="Successfully retrieved
│ EKS cluster details" name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info msg="Syncing Kubernetes
│ Applications to:" name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info
│ msg="	kube-proxy:	1.22.11-minimal-eksbuild.2" name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info msg="	coredns:	1.8.7"
│ name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info msg="	VPC CNI
│ Plugin:	1.11.4" name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info msg="Loading Kubernetes
│ Client" name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info msg="Retrieving details for
│ EKS cluster arn:aws:eks:eu-central-1:xxx:cluster/xxxx"
│ name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info msg="Detected cluster
│ deployed in region eu-central-1" name=kubergrunt
│ [] time="2023-01-09T19:27:54+01:00" level=info msg="Successfully retrieved
│ EKS cluster details" name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Current deployed
│ version matches expected version. Skipping kube-proxy update."
│ name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Confirming
│ compatibility of coredns configuration with latest version."
│ name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Confirming
│ compatibility of coredns permissions with latest version." name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Current deployed
│ version matches expected version. Skipping coredns update." name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Creating temporary file
│ to act as kubeconfig with auth info" name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Created
│ /var/folders/9c/zjqq57kj14j2hr48hy8ldm680000gn/T/3837330752 to act as
│ temporary kubeconfig file." name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Retrieving details for
│ EKS cluster arn:aws:eks:eu-central-1:xxxx:cluster/xxxx"
│ name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Detected cluster
│ deployed in region eu-central-1" name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Successfully retrieved
│ EKS cluster details" name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Appending cluster info
│ for default to kubectl config." name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Successfully appended
│ cluster info." name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Adding auth info to
│ config" name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Done adding auth info
│ to config" name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Appending context
│ default to kubectl config." name=kubergrunt
│ [] time="2023-01-09T19:27:55+01:00" level=info msg="Successfully appended
│ context to kubectl config." name=kubergrunt
│ ERROR: runtime error: invalid memory address or nil pointer dereference
│ 
╵
ERRO[0564] 1 error occurred:
	* exit status 1

The versions are as follows:

• kubegrunt version: v0.10.0
• kubectl versions: Client Version: v1.23.4 AND Server Version: v1.22.16-eks-ffeb93d
• terragrunt version: v0.35.16
• aws version: aws-cli/2.7.31 Python/3.10.7 Darwin/22.1.0 source/arm64 prompt/off

---

Tracked in ticket #109778

[gitsstewart]

Thanks for making us aware of this issue @sewmiuraj . Our team has been working on getting our EKS modules upgraded to 1.24 and may have some insights. We'll take a look this week and update here as we are able.

[sewmiuraj]

@gitsstewart Thank you for letting me know. Once I downgraded my kubegrunt version to v0.9.0, I was able to complete the upgrade for the control plane.

[pras111gw]

@sewmiuraj Thank you for that detailed explanation. I will download 0.95.1 to replicate your situation and answer your question in a couple days.

[pras111gw]

@sewmiuraj In 0.95.1, I see an install.sh script. Are you using that by any chance? If not, please provide your install script here. Thank you.

[sewmiuraj]

@pras111gw Yes, we are using the file install.sh to Install utilities.

The versions we use are as follows:
GRUNTWORK_INSTALLER_VERSION: v0.0.36
MODULE_CI_VERSION: v0.39.6
MODULE_SECURITY_VERSION: v0.55.4

[pras111gw]

@sewmiuraj Perfect! In the v0.95.1 install.sh on line 15 (Build tooling block), we set the version
readonly DEFAULT_KUBERGRUNT_VERSION="v0.9.1"
Please update this line to kubergrunt version of your choice.
Let us know if this resolves your issue, or if you need anything else.

[gorkemgoknar]

Notice for all that use ecs-deploy-runner (reference-architecture)
You must build deploy runner with relevant versions of kubectl and kubergrunt, even on v0.51.1 version kubergrunt is set to 0.8.0 default and you should upgrade it accordingly
https://github.com/gruntwork-io/terraform-aws-ci/blob/de303792e9b54737620f31cae1da6777d4e1f468/modules/ecs-deploy-runner/docker/deploy-runner/Dockerfile#L46