IPV6 bug from using deprecated kubernetes client-node library

[gattytto]

Checks

• I have searched the existing issues.
• My issue is related to one of the components in the kubeflow/dashboard repository.

Kubeflow Version

all

Kubeflow Platform

manifests

Kubernetes Distribution

on premises pure ipv6 calico+cri-o

Kubernetes Version

Client Version: v1.30.13
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.2

Description

according to https://github.com/kubernetes-client/javascript/blob/release-1.x/FETCH_MIGRATION.md kubernetes client-node library has switched out from request module.
Current version of centraldashboard uses a deprecated version of kubernetes/client-node module which fails to resolve a host address in ipv6 format.
This results in central dashboard landing breaking

temprary workaround by editing the centraldashboard deployment:

    spec:
      containers:
      - env:
        - name: KUBERNETES_SERVICE_HOST
          value: kubernetes.default
        - name: KUBERNETES_PORT_443_TCP_ADDR
          value: kubernetes.default

logs before the workaround:

Unable to fetch ConfigMap: Error [ERR_TLS_CERT_ALTNAME_INVALID]: Hostname/IP does not match certificate's altnames: Host: 1101. is not in the cert's altnames: DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:localhost, IP Address:1101:300:1:2:0:0:0:1, IP Address:2001:470:1B1E:C000:216:3EFF:FE45:4D4, IP Address:2001:470:1B1E:C000:0:0:0:1, IP Address:0:0:0:0:0:0:0:1, IP Address:1100:200:0:0:0:0:0:1, IP Address:1101:300:1:2:0:0:0:A

Proposed solution: I don't know enough about nodejs but a start would be to upgrade the client-node reference in all packages (this may affect pipelines and other deployments).

[gattytto]

and then with the workaround applied:

[gattytto]

@andyatmiami thank you very much for you assistance on finding this

[gattytto]

ml-pipeline-ui deployment has similar issue when trying to use pipelines menu options.

kubectl -n kubeflow logs --tail=20 -f deploy/ml-pipeline-ui
GET /pipeline/apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter=
Proxied request:  /:300:1:2::f754/apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter=
[HPM] Error occurred while trying to proxy request /apis/v2beta1/pipelines?page_token=&page_size=10&sort_by=created_at%20desc&filter= from [::1]:666 to http://1101:300:1:2::f754:8888 (ENETUNREACH) (https://nodejs.org/api/errors.html#errors_common_system_errors)

just like previous situation, fix is as follows by modifying the ml-pipeline-ui deployment spec:

spec:
  containers:
  - env:
    - name: ML_PIPELINE_SERVICE_HOST
      value: "ml-pipeline"

[gattytto]

here's the pipelines ui after the change

[gattytto]

for the tensorboards menu I don't see where the error comes from as it doesn't produce any logs in centraldashboard deployment but the issue seems to be similar to previous cases

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity.
It will be closed if no further activity occurs.
Thank you for your contributions.

Members may comment /lifecycle frozen to prevent this issue from being marked as stale.

[gattytto]

/remove-lifecycle stale

[gattytto]

/lifecycle frozen