feat: Implement Training Options pattern for flexible TrainJob customization #91

abhijeet-dhumal · 2025-09-12T13:17:20Z

What this PR does / why we need it:

Sample import styles for Options :

# Style 1: Only works with Kubernetes + common import options
from kubeflow.trainer import (
    Labels, Annotations, SpecLabels, SpecAnnotations,
    Name, PodTemplateOverrides, TrainerImage, TrainerCommand, TrainerArgs,
    PodTemplateOverride, PodTemplateSpecOverride, ContainerOverride
)

# Style 2: Import all options - SDK validates at runtime and rejects incompatible ones
from kubeflow.trainer.options import (
    # Kubernetes options
    Labels, Annotations, SpecLabels, SpecAnnotations, Name,
    PodTemplateOverrides, TrainerImage, TrainerCommand, TrainerArgs,
    # Common options (used in PodTemplateOverrides)
    PodTemplateOverride, PodTemplateSpecOverride, ContainerOverride,
    # LocalProcess options
    ProcessTimeout, WorkingDirectory
)

# Style 3: Backend-specific imports
from kubeflow.trainer.options.kubernetes import (
    Labels, Annotations, SpecLabels, SpecAnnotations, Name,
    PodTemplateOverrides, TrainerImage, TrainerCommand, TrainerArgs
)
from kubeflow.trainer.options.common import (
    PodTemplateOverride, PodTemplateSpecOverride, ContainerOverride
)
from kubeflow.trainer.options.localprocess import (
    ProcessTimeout, WorkingDirectory
)

Job Metadata

# TrainJob resource metadata (for TrainJob level Labels and annotations)
Labels(labels={"team": "ml-platform", "project": "llm"})
Annotations(annotations={"description": "Fine-tuning job", "owner": "user"})
Name(name="my-custom-job-name")

# Derivative JobSet and Jobs labels (for Pod level Labels and annotations)
SpecLabels(labels={"app": "training", "version": "v1.0", "env": "prod"})
SpecAnnotations(annotations={"prometheus.io/scrape": "true", "cost-center": "ai"})

Container runtime customisation :

TrainerImage(image="custom/pytorch:2.0-cuda11.8")
TrainerCommand(command=["python", "train.py"])
TrainerArgs(args=["--epochs", "100", "--lr", "0.001", "--batch-size", "32"])

Pod level overrides:

PodTemplateOverrides(pod_template_overrides=[
    PodTemplateOverride(
        target_jobs=["node", "launcher"],
        metadata={  # Optional: pod-level labels/annotations
            "labels": {"tier": "training"},
            "annotations": {"sidecar.istio.io/inject": "false"}
        },
        spec=PodTemplateSpecOverride(
            service_account_name="training-sa",
            node_selector={"gpu": "true", "zone": "us-west1"},
            affinity={
                "nodeAffinity": {
                    "requiredDuringSchedulingIgnoredDuringExecution": {
                        "nodeSelectorTerms": [{
                            "matchExpressions": [{
                                "key": "gpu-type",
                                "operator": "In",
                                "values": ["nvidia-a100"]
                            }]
                        }]
                    }
                }
            },
            tolerations=[
                {"key": "gpu", "operator": "Exists", "effect": "NoSchedule"},
                {"key": "high-priority", "operator": "Equal", "value": "true", "effect": "NoSchedule"}
            ],
            volumes=[
                {"name": "data", "emptyDir": {}},
                {"name": "model", "persistentVolumeClaim": {"claimName": "model-pvc"}}
            ],
            containers=[
                ContainerOverride(
                    name="node",
                    env=[
                        {"name": "NCCL_DEBUG", "value": "INFO"},
                        {"name": "WANDB_API_KEY", "valueFrom": {"secretKeyRef": {
                            "name": "wandb-secret", "key": "api-key"
                        }}}
                    ],
                    volume_mounts=[
                        {"name": "data", "mountPath": "/data"},
                        {"name": "model", "mountPath": "/model"}
                    ]
                )
            ],
            init_containers=[
                ContainerOverride(
                    name="init-setup",
                    env=[{"name": "SETUP_MODE", "value": "fast"}],
                    volume_mounts=[{"name": "data", "mountPath": "/setup"}]
                )
            ],
            scheduling_gates=[{"name": "wait-for-quota"}],
            image_pull_secrets=[{"name": "registry-credentials"}]
        )
    )
])

Local process backend options:

from kubeflow.trainer.options import ProcessTimeout, WorkingDirectory
ProcessTimeout(timeout_seconds=3600)
WorkingDirectory(working_dir="/custom/training/path")

Complete example :

job_name = client.train(
    runtime=client.get_runtime("torch-distributed"),
    trainer=CustomTrainer(
        func=lambda: print("Training LLM!"),
        num_nodes=2,
        resources_per_node={
            "cpu": "8",
            "memory": "32Gi",
            "nvidia.com/gpu": "2"
        },
        packages_to_install=["torch==2.0.0", "transformers", "datasets"],
        env={"WANDB_PROJECT": "llm-training"}
    ),
    options=[
        # TrainJob resource metadata
        Name(name="llm-finetune-001"),
        Labels(labels={"owner": "ml-team", "project": "llm"}),
        Annotations(annotations={"description": "GPT fine-tuning experiment"}),
        
        # Derivative JobSet/Jobs labels (for training pods)
        SpecLabels(labels={"app": "training", "version": "v1.0", "env": "prod"}),
        SpecAnnotations(annotations={"prometheus.io/scrape": "true"}),
        
        # Trainer container customization
        TrainerImage(image="my-registry/pytorch:2.0-cuda11.8"),
        
        # Pod-level customizations
        PodTemplateOverrides(pod_template_overrides=[
            PodTemplateOverride(
                target_jobs=["node"],
                spec=PodTemplateSpecOverride(
                    service_account_name="gpu-training-sa",
                    node_selector={"accelerator": "nvidia-a100", "zone": "us-west1"},
                    tolerations=[
                        {"key": "nvidia.com/gpu", "operator": "Exists", "effect": "NoSchedule"}
                    ],
                    volumes=[
                        {"name": "data", "persistentVolumeClaim": {"claimName": "training-data-pvc"}},
                        {"name": "cache", "emptyDir": {"sizeLimit": "50Gi"}}
                    ],
                    containers=[
                        ContainerOverride(
                            name="node",
                            env=[
                                {"name": "NCCL_DEBUG", "value": "INFO"},
                                {"name": "NCCL_IB_DISABLE", "value": "1"},
                                {"name": "WANDB_API_KEY", "valueFrom": {
                                    "secretKeyRef": {"name": "wandb-creds", "key": "api-key"}
                                }}
                            ],
                            volume_mounts=[
                                {"name": "data", "mountPath": "/data"},
                                {"name": "cache", "mountPath": "/cache"}
                            ]
                        )
                    ]
                )
            )
        ])
    ]
)

Checklist:

Docs included if any changes are user facing

kubeflow/trainer/options/options.py

szaher

Thanks @abhijeet-dhumal

I believe we need to handle how options will be applied for while backends. either we ignore options for localprocess or make options targeted towards specific backend.

kubeflow/trainer/api/trainer_client.py

kubeflow/trainer/backends/kubernetes/backend.py

coveralls · 2025-09-24T11:42:32Z

Pull Request Test Coverage Report for Build 18870799839

Details

37 of 37 (100.0%) changed or added relevant lines in 2 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+2.7%) to 76.11%

Files with Coverage Reduction	New Missed Lines	%
kubeflow/trainer/utils/utils.py	1	70.09%

Totals
Change from base Build 18826380320:	2.7%
Covered Lines:	360
Relevant Lines:	473

💛 - Coveralls

…bSet/Jobs labels Signed-off-by: Abhijeet Dhumal <[email protected]>

astefanutti

Left a few comments, otherwise looks good to me.

kubeflow/trainer/api/trainer_client.py

kubeflow/trainer/types/types.py

kubeflow/trainer/options/kubernetes.py

andreyvelich · 2025-10-27T00:05:55Z

kubeflow/trainer/options/__init__.py

+from kubeflow.trainer.options.common import (
+    ContainerOverride,
+    PodTemplateOverride,
+    PodTemplateSpecOverride,
+)
+from kubeflow.trainer.options.kubernetes import (
+    Annotations,
+    Labels,
+    Name,
+    PodTemplateOverrides,
+    SpecAnnotations,


Can we only start with PodTemplateOverrides options and remove other options for now?
We can introduce them later, if users require them.

Suggested change

from kubeflow.trainer.options.common import (

ContainerOverride,

PodTemplateOverride,

PodTemplateSpecOverride,

)

from kubeflow.trainer.options.kubernetes import (

Annotations,

Labels,

Name,

PodTemplateOverrides,

SpecAnnotations,

from kubeflow.trainer.options.kubernetes import (

Annotations,

Labels,

Name,

PodTemplateOverrides,

SpecAnnotations,

Actually If we planning to implement podtemplateOverrides
then we need all three because they're direct mappings to the TrainJob CRD schema (verified against the latest trainer.kubeflow.org_trainjobs.yaml).
IMHO We can't flatten this - the nested structure is required by the API, and each class serves a distinct purpose: targeting (PodTemplateOverride), pod-level config (PodTemplateSpecOverride), and container-specific overrides (ContainerOverride).

The hierarchy mirrors TrainJob pod structure:

PodTemplateOverride targets specific job types (e.g., "node" vs "launcher"),

PodTemplateSpecOverride handles pod-level configs (serviceAccountName, nodeSelector, affinity, tolerations, volumes, imagePullSecrets, schedulingGates),

ContainerOverride lets users modify specific containers by name (env vars, volume mounts).

Without these, users can't do basic production scenarios like mounting datasets from PVCs, scheduling on GPU nodes, pulling from private registries, or injecting secrets as env vars. The CRD explicitly defines ContainerOverride with name/env/volumeMounts fields, and spec.containers and spec.initContainers are lists of ContainerOverride objects.

These are essential building blocks, not optional nice-to-haves.

# Mount dataset from PVC, add GPU affinity PodTemplateOverride( target_jobs=["node"], spec=PodTemplateSpecOverride( node_selector={"gpu": "a100"}, volumes=[{"name": "data", "persistentVolumeClaim": {"claimName": "dataset-pvc"}}], containers=[ ContainerOverride( name="trainer", env=[{"name": "HF_TOKEN", "valueFrom": {"secretKeyRef": ...}}], volume_mounts=[{"name": "data", "mountPath": "/data"}] ) ] ) )

WDYT? @andreyvelich @astefanutti

I'd be happy to discuss if I've misunderstood you!

Yes that makes sense, I think that's acceptable. The other alternative would be to use dictionaries, but that looks less elegant.

andreyvelich · 2025-10-27T00:06:35Z

kubeflow/trainer/api/trainer_client_test.py

+from kubeflow.trainer.types import types
+
+
+class TestTrainerClientOptionValidation:


This is still open @abhijeet-dhumal.

kubeflow/trainer/options/kubernetes.py

kubeflow/trainer/options/common.py

kubeflow/trainer/options/kubernetes.py

astefanutti · 2025-10-27T11:16:59Z

kubeflow/trainer/types/types.py

+
+    def __call__(
+        self,
+        job_spec: dict,


Should be type it to TrainerV1alpha1TrainJob or we keep it flexible?

I think we can keep it flexible here ?

job_spec = {} if options: for option in options: option(job_spec, trainer)

The options creates a plain dict that represents the TrainJob structure. The backend then extracts values from this dict and constructs the actual models.TrainerV1alpha1TrainJob object later. So that way backends can handle conversion to their specific types

I'd be happy to discuss if I've misunderstood you!

That sounds good, let's start by being flexible and iterate if needed.

Signed-off-by: Abhijeet Dhumal <[email protected]>

abhijeet-dhumal · 2025-10-27T15:50:11Z

#91 (comment)

This is still open @abhijeet-dhumal.

@andreyvelich I don't understand this point I'm afraid 😅
Could you please elaborate it a bit more

Signed-off-by: Abhijeet Dhumal <[email protected]>

scripts/gen-changelog.py

astefanutti · 2025-10-27T18:40:07Z

/lgtm

Thanks @abhijeet-dhumal for this awesome contribution!

Signed-off-by: Abhijeet Dhumal <[email protected]>

astefanutti · 2025-10-28T10:05:54Z

@abhijeet-dhumal Thanks!

/lgtm
/approve

/hold for @andreyvelich approval

google-oss-prow · 2025-10-28T10:06:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astefanutti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [astefanutti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bohdandenysenko · 2025-10-28T10:13:33Z

kubeflow/trainer/backends/kubernetes/backend.py

+            metadata=models.IoK8sApimachineryPkgApisMetaV1ObjectMeta(
+                name=train_job_name, labels=labels, annotations=annotations
+            ),
            spec=models.TrainerV1alpha1TrainJobSpec(


@abhijeet-dhumal, awesome, looks good to me now!

google-oss-prow · 2025-10-28T10:13:48Z

@bohdandenysenko: changing LGTM is restricted to collaborators

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kramaranya · 2025-10-28T10:14:57Z

/hold
Sorry, will review it today!

andreyvelich · 2025-10-28T13:46:47Z

@abhijeet-dhumal As I suggested here: https://github.com/kubeflow/sdk/pull/91/files#r2412261172, can you refactor unit tests: https://github.com/kubeflow/sdk/blob/d32626e4390cd2f9cc25407b88a82bd1cf09973a/kubeflow/trainer/api/trainer_client_test.py ?

The test cases for train API in Kubernetes backend can go to:

sdk/kubeflow/trainer/backends/kubernetes/backend_test.py

Line 1502 in d32626e

def test_train(kubernetes_backend, test_case):

Similar to Kubernetes, we can create unit tests for train() API in local subprocess: https://github.com/kubeflow/sdk/blob/d32626e4390cd2f9cc25407b88a82bd1cf09973a/kubeflow/trainer/backends/localprocess/backend_test.py

kramaranya

Thank you @abhijeet-dhumal for this!
I left a few comments

kramaranya · 2025-10-28T18:23:30Z

kubeflow/trainer/__init__.py

+    "Name",
+    "PodTemplateOverride",
+    "PodTemplateOverrides",
+    "PodTemplateSpecOverride",
    "Runtime",
+    "RuntimeTrainer",
    "TorchTuneConfig",
    "TorchTuneInstructDataset",
-    "RuntimeTrainer",
+    "TrainerArgs",
    "TrainerClient",
+    "TrainerCommand",
+    "TrainerImage",
    "TrainerType",
-    "LocalProcessBackendConfig",
-    "KubernetesBackendConfig",


SpecLabels and SpecAnnotations are missing

kramaranya · 2025-10-28T18:35:06Z

kubeflow/trainer/api/trainer_client.py

+            options: Optional list of configuration options to apply to the TrainJob. Use
+                Labels and Annotations for basic metadata configuration.


I'd suggest also mentioning container and pod options

kramaranya · 2025-10-28T21:56:16Z

kubeflow/trainer/backends/kubernetes/backend.py

+        # Generate unique name for the TrainJob if not provided
+        train_job_name = name or (
+            random.choice(string.ascii_lowercase)
+            + uuid.uuid4().hex[: constants.TRAINJOB_NAME_UUID_LENGTH]
+        )


What if the name user provides is not unique?

kramaranya · 2025-10-28T22:09:27Z

kubeflow/trainer/backends/kubernetes/backend.py

-                f"Timeout to get {constants.CLUSTER_TRAINING_RUNTIME_PLURAL}: "
-                f"{self.namespace}/{name}"
+                f"Timeout to get {constants.CLUSTER_TRAINING_RUNTIME_PLURAL}: {name}"


Since you dropped namespace here, could you also update other instances? For example in list_runtimes https://github.com/kubeflow/sdk/blob/main/kubeflow/trainer/backends/kubernetes/backend.py#L90-L98

kramaranya · 2025-10-28T22:34:56Z

kubeflow/trainer/options/common.py

+    def __call__(
+        self,
+        job_spec: dict[str, Any],
+        trainer: BuiltinTrainer | CustomTrainer | None = None,


This requires python 3.10, since we still support 3.9, could you use optional/union instead?
https://peps.python.org/pep-0604/

kramaranya · 2025-10-28T22:36:26Z

kubeflow/trainer/options/kubernetes.py

+    env: list[dict] | None = None
+    volume_mounts: list[dict] | None = None


Same here, please use optional/union

kramaranya · 2025-10-28T22:39:51Z

kubeflow/trainer/types/types.py

-    func: Callable
-    func_args: Optional[dict] = None
-    packages_to_install: Optional[list[str]] = None
+    func: Callable | None = None
+    func_args: dict | None = None
+    packages_to_install: list[str] | None = None


Same here, please revert these changes where needed

kramaranya · 2025-10-28T22:46:59Z

kubeflow/trainer/types/types.py

-TORCH_TUNE = BuiltinTrainer.__annotations__["config"].__name__.lower().replace("config", "")
+TORCH_TUNE = BuiltinTrainer.__annotations__["config"].lower().replace("config", "")


What's the reason behind this change?

kramaranya · 2025-10-28T22:49:41Z

pyproject.toml

  "kubernetes>=27.2.0",
  "pydantic>=2.10.0",
  "kubeflow-trainer-api>=2.0.0",
+  "eval-type-backport>=0.2.0; python_version < '3.10'",


Why do we need this?

I think that provides retro-compatibility for union types.

andreyvelich · 2025-10-30T17:56:49Z

@abhijeet-dhumal Did you get a chance to address this comment: #91 (comment), and @kramaranya suggestions ?

google-oss-prow bot added the do-not-merge/work-in-progress label Sep 12, 2025

google-oss-prow bot requested review from kramaranya and szaher September 12, 2025 13:17

google-oss-prow bot added the size/XL label Sep 12, 2025

abhijeet-dhumal changed the title ~~Add labels and annotations support for train client~~ feast: Add labels and annotations support for train client Sep 12, 2025

abhijeet-dhumal changed the title ~~feast: Add labels and annotations support for train client~~ feat: Add labels and annotations support for train client Sep 15, 2025

abhijeet-dhumal force-pushed the add-labels-annotations-support#87 branch from 67d12d8 to b39b364 Compare September 15, 2025 05:22

google-oss-prow bot added size/XXL and removed size/XL labels Sep 15, 2025

abhijeet-dhumal force-pushed the add-labels-annotations-support#87 branch 2 times, most recently from 2d7a8e6 to dbba135 Compare September 15, 2025 05:44

abhijeet-dhumal mentioned this pull request Sep 15, 2025

Expose TrainJob labels and annotations in the SDK #87

Open

abhijeet-dhumal changed the title ~~feat: Add labels and annotations support for train client~~ feat: Implement Training Options pattern with WithLabels, WithAnnotations, and WithPodSpecOverrides for flexible TrainJob customization Sep 15, 2025

abhijeet-dhumal changed the title ~~feat: Implement Training Options pattern with WithLabels, WithAnnotations, and WithPodSpecOverrides for flexible TrainJob customization~~ feat: Implement Training Options pattern for flexible TrainJob customization Sep 15, 2025

astefanutti reviewed Sep 15, 2025

View reviewed changes

kubeflow/trainer/options/options.py Outdated Show resolved Hide resolved

abhijeet-dhumal force-pushed the add-labels-annotations-support#87 branch from 36c0160 to 95155f6 Compare September 15, 2025 10:41

abhijeet-dhumal requested a review from astefanutti September 15, 2025 10:43

abhijeet-dhumal marked this pull request as ready for review September 15, 2025 11:23

google-oss-prow bot removed the do-not-merge/work-in-progress label Sep 15, 2025

szaher suggested changes Sep 15, 2025

View reviewed changes

kubeflow/trainer/api/trainer_client.py Outdated Show resolved Hide resolved

kubeflow/trainer/backends/kubernetes/backend.py Outdated Show resolved Hide resolved

google-oss-prow bot assigned szaher Sep 15, 2025

astefanutti reviewed Sep 15, 2025

View reviewed changes

kubeflow/trainer/backends/kubernetes/backend.py Outdated Show resolved Hide resolved

kubeflow/trainer/backends/kubernetes/backend.py Outdated Show resolved Hide resolved

abhijeet-dhumal marked this pull request as draft September 17, 2025 13:17

google-oss-prow bot added the do-not-merge/work-in-progress label Sep 17, 2025

abhijeet-dhumal force-pushed the add-labels-annotations-support#87 branch 3 times, most recently from f092845 to 03994e7 Compare September 24, 2025 11:40

abhijeet-dhumal requested review from astefanutti and szaher September 24, 2025 12:05

abhijeet-dhumal requested a review from bohdandenysenko October 23, 2025 13:12

feat: add spec-level labels and annotations options for derivative Jo…

5a7c563

…bSet/Jobs labels Signed-off-by: Abhijeet Dhumal <[email protected]>

astefanutti reviewed Oct 24, 2025

View reviewed changes

kubeflow/trainer/api/trainer_client.py Outdated Show resolved Hide resolved

kubeflow/trainer/types/types.py Outdated Show resolved Hide resolved

kubeflow/trainer/options/kubernetes.py Outdated Show resolved Hide resolved

andreyvelich reviewed Oct 27, 2025

View reviewed changes

astefanutti reviewed Oct 27, 2025

View reviewed changes

abhijeet-dhumal force-pushed the add-labels-annotations-support#87 branch from 338ed1c to 938b322 Compare October 27, 2025 15:16

fix: Minor improvemnts in imports and trainer types

b1ce0f5

Signed-off-by: Abhijeet Dhumal <[email protected]>

abhijeet-dhumal force-pushed the add-labels-annotations-support#87 branch from 938b322 to b1ce0f5 Compare October 27, 2025 15:27

abhijeet-dhumal added 2 commits October 27, 2025 21:29

fix: Adjusted PodtemplateOverrides userflow for SDK

7a09dc9

Signed-off-by: Abhijeet Dhumal <[email protected]>

fix: Skip/TODO options for Container and LocalProcess backends

b229a5c

Signed-off-by: Abhijeet Dhumal <[email protected]>

abhijeet-dhumal requested review from andreyvelich and astefanutti October 27, 2025 16:14

astefanutti reviewed Oct 27, 2025

View reviewed changes

scripts/gen-changelog.py Show resolved Hide resolved

google-oss-prow bot assigned astefanutti Oct 27, 2025

google-oss-prow bot added the lgtm label Oct 27, 2025

fix: Make gen-changelog.py file executable

d32626e

Signed-off-by: Abhijeet Dhumal <[email protected]>

google-oss-prow bot removed the lgtm label Oct 28, 2025

google-oss-prow bot added do-not-merge/hold lgtm labels Oct 28, 2025

google-oss-prow bot added the approved label Oct 28, 2025

bohdandenysenko approved these changes Oct 28, 2025

View reviewed changes

kramaranya reviewed Oct 28, 2025

View reviewed changes

		from kubeflow.trainer.types import types


		class TestTrainerClientOptionValidation:

		options: Optional list of configuration options to apply to the TrainJob. Use
		Labels and Annotations for basic metadata configuration.

		env: list[dict] \| None = None
		volume_mounts: list[dict] \| None = None

		TORCH_TUNE = BuiltinTrainer.__annotations__["config"].__name__.lower().replace("config", "")
		TORCH_TUNE = BuiltinTrainer.__annotations__["config"].lower().replace("config", "")

feat: Implement Training Options pattern for flexible TrainJob customization #91

Are you sure you want to change the base?

feat: Implement Training Options pattern for flexible TrainJob customization #91

Uh oh!

Conversation

abhijeet-dhumal commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

szaher left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 18870799839

Details

💛 - Coveralls

Uh oh!

astefanutti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhijeet-dhumal commented Oct 27, 2025

Uh oh!

Uh oh!

astefanutti commented Oct 27, 2025

Uh oh!

astefanutti commented Oct 28, 2025

Uh oh!

google-oss-prow bot commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented Oct 28, 2025

Uh oh!

kramaranya commented Oct 28, 2025

Uh oh!

andreyvelich commented Oct 28, 2025

Uh oh!

kramaranya left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

abhijeet-dhumal commented Sep 12, 2025 •

edited

Loading

coveralls commented Sep 24, 2025 •

edited

Loading