Skip to content

Running the sample workflow for data transformation results in a workflow stuck in "Running" status #14990

New issue
New issue
Open
Bug
Open
Running the sample workflow for data transformation results in a workflow stuck in "Running" status#14990
Bug
@chaicheehwee

Description

@chaicheehwee
chaicheehwee
opened on Oct 26, 2025

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Running the sample code for data transformation from
https://raw.githubusercontent.com/argoproj/argo-workflows/refs/heads/main/examples/data-transformations.yaml
results in a workflow that is stuck in "Running" status.
The "list-log-files" step is stuck in "Pending" status

Logs from the workflow's main container indicate a missing file/folder:

time="2025-10-26T14:05:26 UTC" level=info msg="capturing logs" argo=true
time="2025-10-26T14:05:26 UTC" level=info msg="Starting Workflow Executor" version=v3.7.2
time="2025-10-26T14:05:26 UTC" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-10-26T14:05:26 UTC" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=true namespace=argo podName=data-transformations-tr66k-list-log-files-1285757969 templateName=list-log-files version="&Version{Version:v3.7.2,BuildDate:2025-09-11T14:56:36Z,GitCommit:cf605b548f1b0137836dde2032517707ba87c6ae,GitTag:v3.7.2,GitTreeState:clean,GoVersion:go1.24.4,Compiler:gc,Platform:linux/amd64,}"
time="2025-10-26T14:05:26 UTC" level=info msg="Creating minio client using static credentials" endpoint="minio.dev.me:32000"
time="2025-10-26T14:05:26 UTC" level=info msg="Listing directory from s3" bucket=my-bucket endpoint="minio.dev.me:32000" key=
time="2025-10-26T14:05:26 UTC" level=info msg="List objects" artifactName=test-bucket duration=12.420562ms error="<nil>" key=
time="2025-10-26T14:05:27 UTC" level=info msg="sub-process exited" argo=true error="<nil>"
time="2025-10-26T14:05:27 UTC" level=warning msg="cannot save artifact /file" argo=true error="stat /file: no such file or directory"

The work-around when "archiveLogs" for the artifact repository is true is to remove this section:

      outputs:
        artifacts:
          - name: file
            path: /file

The work-around when "archiveLogs" for the artifact repository is false is to set optional to true:

      outputs:
        artifacts:
          - name: file
            path: /file
            optional: true

After asking about this in the "Discussions" #14940 , it seems like this is a bug.

Version(s)

v3.7.3,v3.7.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

# The workflow is from https://raw.githubusercontent.com/argoproj/argo-workflows/refs/heads/main/examples/data-transformations.yaml
# See doc docs/data-sourcing-and-transformation.md
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-transformations-
  annotations:
    workflows.argoproj.io/description: |
      This workflow demonstrates using a data template to list in an S3 bucket
      and then process those log files.
    workflows.argoproj.io/version: ">= 3.1.0"
spec:
  entrypoint: data-transformations
  templates:
    - name: data-transformations
      steps:
        - - name: list-log-files
            template: list-log-files
        - - name: process-logs
            template: process-logs
            withParam: "{{steps.list-log-files.outputs.result}}"
            arguments:
              artifacts:
                - name: file
                  s3:
                    key: "{{item}}"

              parameters:
                - name: file-name
                  value: "{{item}}"

    - name: list-log-files
      data:
        source:
          artifactPaths:
            name: test-bucket
            s3:
              bucket: my-bucket

        transformation:
          - expression: "filter(data, {# endsWith \"main.log\"})"
      outputs:
        artifacts:
          - name: file
            path: /file

    - name: process-logs
      inputs:
        parameters:
          - name: file-name
        artifacts:
          - name: file
            path: /file
      container:
        image:  argoproj/argosay:v2
        command: [ sh, -c ]
        args:
          - |
            echo {{inputs.parameters.file-name}}
            head /file
---
# ConfigMap used for toggling the "archiveLogs" field:
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: argo
  name: artifact-repositories
  annotations:
    workflows.argoproj.io/default-artifact-repository: minio-artifact-repository
data:
  minio-artifact-repository: |
    archiveLogs: true
    s3:
      bucket: my-bucket
      endpoint: minio.dev.me:32000
      insecure: true
      accessKeySecret:
        name: my-minio-creds
        key: accesskey
      secretKeySecret:
        name: my-minio-creds
        key: secretkey

Logs from the workflow controller

time="2025-10-26T14:05:24.629Z" level=info msg="Processing workflow" Phase= ResourceVersion=505130 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.641Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.641Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.641Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.641Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.641Z" level=info msg="Steps node data-transformations-tr66k initialized Running" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.641Z" level=info msg="StepGroup node data-transformations-tr66k-2804801334 initialized Running" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.642Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.642Z" level=info msg="Pod node data-transformations-tr66k-1285757969 initialized Pending" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.677Z" level=info msg="Created pod: data-transformations-tr66k[0].list-log-files (data-transformations-tr66k-list-log-files-1285757969)" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.677Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.678Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.678Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.681Z" level=info msg="add pod event" pod=data-transformations-tr66k-list-log-files-1285757969
time="2025-10-26T14:05:24.718Z" level=info msg="update pod event" pod=data-transformations-tr66k-list-log-files-1285757969
time="2025-10-26T14:05:24.761Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=505135 workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.762Z" level=info msg="update pod event" pod=data-transformations-tr66k-list-log-files-1285757969
time="2025-10-26T14:05:24.768Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505135 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.768Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.769Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.769Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.769Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.784Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=505139 workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.785Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505139 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.785Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.786Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.786Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:24.786Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:26.026Z" level=info msg="update pod event" pod=data-transformations-tr66k-list-log-files-1285757969
time="2025-10-26T14:05:27.114Z" level=info msg="update pod event" pod=data-transformations-tr66k-list-log-files-1285757969
time="2025-10-26T14:05:28.228Z" level=info msg="update pod event" pod=data-transformations-tr66k-list-log-files-1285757969
time="2025-10-26T14:05:29.691Z" level=info msg="update pod event" pod=data-transformations-tr66k-list-log-files-1285757969
time="2025-10-26T14:05:34.710Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505139 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.710Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.710Z" level=info msg="leaving phase un-changed: required outputs are not yet set" namespace=argo new.phase=Succeeded workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.712Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.712Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.712Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.733Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=505173 workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.738Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505173 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.739Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.739Z" level=info msg="leaving phase un-changed: required outputs are not yet set" namespace=argo new.phase=Succeeded workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.740Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.741Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:34.741Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:44.736Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505173 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:44.736Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=data-transformations-tr66k
time="2025-10-26T14:05:44.736Z" level=info msg="leaving phase un-changed: required outputs are not yet set" namespace=argo new.phase=Succeeded workflow=data-transformations-tr66k
time="2025-10-26T14:05:44.738Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:44.738Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:05:44.738Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:08:51.465Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505173 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:08:51.465Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=data-transformations-tr66k
time="2025-10-26T14:08:51.466Z" level=info msg="leaving phase un-changed: required outputs are not yet set" namespace=argo new.phase=Succeeded workflow=data-transformations-tr66k
time="2025-10-26T14:08:51.467Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:08:51.467Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:08:51.467Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:17:27.481Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505173 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:17:27.482Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=data-transformations-tr66k
time="2025-10-26T14:17:27.483Z" level=info msg="leaving phase un-changed: required outputs are not yet set" namespace=argo new.phase=Succeeded workflow=data-transformations-tr66k
time="2025-10-26T14:17:27.486Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:17:27.487Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:17:27.487Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:23:39.242Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505173 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:23:39.242Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=data-transformations-tr66k
time="2025-10-26T14:23:39.242Z" level=info msg="leaving phase un-changed: required outputs are not yet set" namespace=argo new.phase=Succeeded workflow=data-transformations-tr66k
time="2025-10-26T14:23:39.243Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:23:39.243Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:23:39.243Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:24:22.468Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=505173 namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:24:22.468Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=data-transformations-tr66k
time="2025-10-26T14:24:22.469Z" level=info msg="leaving phase un-changed: required outputs are not yet set" namespace=argo new.phase=Succeeded workflow=data-transformations-tr66k
time="2025-10-26T14:24:22.470Z" level=info msg="Workflow step group node data-transformations-tr66k-2804801334 not yet completed" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:24:22.471Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=data-transformations-tr66k
time="2025-10-26T14:24:22.471Z" level=info msg=reconcileAgentPod namespace=argo workflow=data-transformations-tr66k

Logs from in your workflow's wait container

error: container wait is not valid for pod data-transformations-tr66k-list-log-files-128575796

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions