KIC 3.5 startup failed no endpoints for service: error

[cello86]

Hi all,

I’m trying to installa KIC 3.5 o a AKS cluster and I use the chart kong/kong to create 2 separated deployment. The first deployment is used to deploy only the kong gateway/proxy and configure Azure load balancer. The endpoints are correctly present behind the healess service:

kubectl get endpoints -n ingress ingress-kong-proxy-kong-admin
Warning: v1 Endpoints is deprecated in v1.33+; use ``discovery.k8s.io/v1`` EndpointSlice
NAME                            ENDPOINTS                             AGE
ingress-kong-proxy-kong-admin   10.244.4.172:8001,10.244.5.107:8001   103m

The second deployment is used to deploy the ingress controller part and the pod fails the startup with this issue:

2025-09-26T17:20:07Z    error   setup.initialize-kong-clients   Failed to create kong client(s), retrying…    {“delay”: 1, “error”: “no endpoints for service: "ingress/ingress-kong-proxy-kong-admin"”}
2025-09-26T17:20:08Z    error   setup.initialize-kong-clients   Failed to create kong client(s), retrying…    {“delay”: 1, “error”: “no endpoints for service: "ingress/ingress-kong-proxy-kong-admin"”}
2025-09-26T17:20:09Z    error   setup.initialize-kong-clients   Failed to create kong client(s), retrying…    {“delay”: 1, “error”: “no endpoints for service: "ingress/ingress-kong-proxy-kong-admin"”}

I suspect that the serviceaccount created from the helm chart hasn’t the correct permissions but I also configured the RBAC under the ingressController node.
`
Could someone help me to identify the root cause of this issue?

Thanks,

Marcello

[cello86]

I'm trying to debug deeper the issue and I installed KIC 3.5 in sidecar mode and it worked fine. I also tried to install KIC in 2 different deployment and disable the gateway discovery on the controller and point to the admin service via service dns and also in this configuration worked fine. I cannot use the service DNS because the configuration update from the controller will be deployer on a single gateway pod.
I noticed that the serviceaccount used on the controller didn't have the permission to access only to the endpointslices and not endpoints and I tried to force the permissions with these manifests:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kong-controller-endpoints-access
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kong-controller-endpoints-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kong-controller-endpoints-access
subjects:
- kind: ServiceAccount
  name: ingress-kong-controller-kong
  namespace: ingress

I had no luck with this change.

Marcello

[cello86]

I also tried to apply the parameter publishNotReadyAddresses as reported on #5110 and change the CONTROLLER_GATEWAY_DISCOVERY_DNS_STRATEGY to"pod" without successo.

I tried the deployment with versions 3.1, 3.2, 3.3 and 3.4 without success.

I tested the serviceaccount from the kong proxy and the endpoints and endpointslices reported ready gateway/proxy pods:

# kubectl proxy --port=8001 &
# curl -s "http://localhost:8001/api/v1/namespaces/ingress/endpoints/ingress-kong-proxy-kong-admin" | jq '.subsets[].addresses'
[
  {
    "ip": "10.244.4.244",
    "nodeName": "aks-infra-24197528-vmss000001",
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-f9vr2",
      "uid": "c15f05a0-acf6-40b9-a80e-a7b2a6f14fb0"
    }
  },
  {
    "ip": "10.244.5.43",
    "nodeName": "aks-infra-24197528-vmss000000",
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-xzch6",
      "uid": "9724ab7c-d015-412a-8dc2-29c55d15286a"
    }
  }
# curl -s "http://localhost:8001/apis/discovery.k8s.io/v1/namespaces/ingress/endpointslices?labelSelector=kubernetes.io%2Fservice-name%3Dingress-kong-proxy-kong-admin" | jq '.items[].endpoints'
[
  {
    "addresses": [
      "10.244.5.43"
    ],
    "conditions": {
      "ready": true,
      "serving": true,
      "terminating": false
    },
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-xzch6",
      "uid": "9724ab7c-d015-412a-8dc2-29c55d15286a"
    },
    "nodeName": "aks-infra-24197528-vmss000000",
    "zone": "westeurope-2"
  },
  {
    "addresses": [
      "10.244.4.244"
    ],
    "conditions": {
      "ready": true,
      "serving": true,
      "terminating": false
    },
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-f9vr2",
      "uid": "c15f05a0-acf6-40b9-a80e-a7b2a6f14fb0"
    },
    "nodeName": "aks-infra-24197528-vmss000001",
    "zone": "westeurope-3"
  }
]
# curl -s "http://localhost:8001/api/v1/namespaces/ingress/endpoints/ingress-kong-proxy-kong-admin" | jq '.subsets[].addresses | length'
2
# curl -s "http://localhost:8001/apis/discovery.k8s.io/v1/namespaces/ingress/endpointslices?labelSelector=kubernetes.io%2Fservice-name%3Dingress-kong-proxy-kong-admin" | jq '.items[].endpoints | length'
2

Marcello

[cello86]

I reproduced the issue with kong ingress controller 3.1.0 and later versions but with 3.0.2 works correctly:

$ kubectl logs -n ingress -f kong-test-controller-6dcb474c5b-qx7wp
2025-10-15T11:55:59Z    info    Diagnostics server disabled     {"v": 0}
2025-10-15T11:55:59Z    info    setup   Starting controller manager     {"v": 0, "release": "3.0.2", "repo": "https://github.com/Kong/kubernetes-ingress-controller.git", "commit": "3386f9698bbde07349e57ad16885165328dab08a"}
2025-10-15T11:55:59Z    info    setup   The ingress class name has been set     {"v": 0, "value": "kong-test"}
2025-10-15T11:55:59Z    info    setup   Getting enabled options and features    {"v": 0}
2025-10-15T11:55:59Z    info    setup   Found configuration option for gated feature    {"v": 0, "feature": "FillIDs", "enabled": true}
2025-10-15T11:55:59Z    info    setup   Getting the kubernetes client configuration     {"v": 0}
W1015 11:55:59.984950       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2025-10-15T11:55:59Z    info    setup   Starting standalone health check server {"v": 0}
2025-10-15T11:55:59Z    info    setup   Getting the kong admin api client configuration {"v": 0}
W1015 11:55:59.985131       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.