PodDefaults manifests incorrectly use vars for Certificate CR

[kimwnasptd]

Checks

• I have searched the existing issues.
• My issue is related to one of the components in the kubeflow/dashboard repository.

Kubeflow Version

Dashboard V2, after the renaming, in master branch

Kubeflow Platform

any

Kubernetes Distribution

any

Kubernetes Version

any

Description

We bumped into this while working on the migration script #154 (comment)

We had initially PR #94 which aimed to replace vars with replacements, since vars are being deprecated in kustomize. But we missed to also update the Certificate CR in the overlays/cert-manager that is responsible for the PodDefault Webhook's certificate (used by K8s to talk to the webhook with https)

dashboard/components/poddefaults-webhooks/manifests/overlays/cert-manager/certificate.yaml

Lines 7 to 10 in 6eb7103

	commonName: $(podDefaultsServiceName).$(podDefaultsNamespace).svc
	dnsNames:
	- $(podDefaultsServiceName).$(podDefaultsNamespace).svc
	- $(podDefaultsServiceName).$(podDefaultsNamespace).svc.cluster.local

This results in the created certificate to have a Common Name of $(podDefaultsServiceName).$(podDefaultsNamespace).svc (the vars are not updated), and thus K8s fails to talk with https as the common name doesn't match the DNS name poddefaults-webhook-service.kubeflow.svc.

Relevant Logs

statefulset/test: create Pod test-0 in StatefulSet test failed error: Internal error occurred: failed calling webhook "deployment.kubeflow.org": failed to call webhook: Post "https://poddefaults-webhook-service.kubeflow.svc:443/apply-poddefault?timeout=10s": tls: failed to verify certificate: x509: certificate is valid for $(podDefaultsServiceName).$(podDefaultsNamespace).svc, $(podDefaultsServiceName).$(podDefaultsNamespace).svc.cluster.local, not poddefaults-webhook-service.kubeflow.svc

[kimwnasptd]

/assign @kimwnasptd

[kimwnasptd]

The main challenge here is that the we need to create a string that relies on different fields.

I'll try to use the delimiter and index properties of kustomize to change the different fields, and use replacements.
https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/#delimiter