chore(ci): updating k8s version in the CI workflows

[nsingla]

Summary

This PR updates Kubernetes versions across CI workflows and associated manifests to modernize the testing infrastructure and ensure compatibility with current Kubernetes releases.

Motivation

• Ensure CI/CD pipelines are testing against supported Kubernetes versions
• Maintain compatibility with current Kubernetes API versions and deprecation cycles
• Improve reliability and security of the testing infrastructure

Changes Made

• CI Workflows: Updated Kubernetes versions in workflow configurations (59 files modified)
• Manifest Updates: Updated deployment configurations, DNS settings, and API server environments
• Configuration Flexibility: Moved hardcoded namespace values to configurable variables to support deployment flexibility
• Infrastructure Modernization: Aligned testing infrastructure with current Kubernetes ecosystem standards

Testing

• CI workflows execute successfully with updated Kubernetes versions
• All existing functionality remains intact across different namespace configurations
• No breaking changes to deployment procedures

Breaking Changes

None - this is a maintenance update focused on CI infrastructure.

Migration Notes

• Namespace configurations are now variable-based rather than hardcoded, improving deployment flexibility
• No action required for existing deployments

Additional Context

This large-scale update (+830 −498 lines across 59 files) ensures the Kubeflow Pipelines project maintains compatibility with modern Kubernetes versions while improving the flexibility of deployment configurations.

Checklist:

• You have signed off your commits
• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign droctothorpe for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mprahl]

Does this prevent deploying to other namespaces?

[nsingla]

good point, i will replace the hard coded namespace with a variable one

[nsingla]

Done, moved to a config file