Skip to content

Arifact GC not executing when a workflow has tasks with inline templates #14922

New issue
New issue
Open
Bug
#14926
Open
Arifact GC not executing when a workflow has tasks with inline templates#14922
Bug
#14926
Labels
area/gcGarbage collection, such as TTLs, retentionPolicy, delays, and more
@NeelArle

Description

@NeelArle
NeelArle
opened on Oct 9, 2025

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

have not checked the latest version of Argo because I don't have control over our environment

Happened:
The artifactGC task was not created and the artifacts remained in a GCS bucket

Expected:
The artifactGC task was created for a workflow with inline templates, artifacts were deleted from a GCS bucket

Version(s)

3.7.0

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

Dear all,

here the manifest i run through Argo Workflow

---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  namespace: alida # you could change it as you want
  generateName: artifact-passing-
spec:
  serviceAccountName: argo-workflow
  entrypoint: artifact-example
  artifactGC:
    strategy: OnWorkflowCompletion 
    serviceAccountName: argo-workflow
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        inline:
          container:
            image: argoproj/argosay:v2
            command: [sh, -c]
            args:
              - |
                echo "hello world" > /tmp/hello_world.txt
          outputs:
            artifacts:
            - name: hello-art
              path: /tmp/hello_world.txt
              artifactGC:
                strategy: OnWorkflowCompletion 
    - - name: consume-artifact
        inline:
          inputs:
            artifacts:
            - name: message
              path: /tmp/message
          container:
            image: alpine:latest
            command: [sh, -c]
            args: ["cat /tmp/message"]
        arguments:
          artifacts:
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"
---

The Argo workflow has been completed, but I can still download the artifact produced by Argo UI.
The artifact is not deleted as expected by the strategy used.

Thanks in advance for your support.

Logs from the workflow controller

time="2025-10-09T14:08:24.320Z" level=info msg="Processing workflow" Phase= ResourceVersion=4270988 namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.458Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2025-10-09T14:08:24.458Z" level=info msg="Task-result reconciliation" namespace=alida numObjs=0 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.458Z" level=info msg="Updated phase  -> Running" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.459Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.460Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.460Z" level=info msg="Steps node artifact-passing-lfcp7 initialized Running" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.461Z" level=info msg="StepGroup node artifact-passing-lfcp7-3140360656 initialized Running" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.461Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.461Z" level=info msg="Pod node artifact-passing-lfcp7-4182222514 initialized Pending" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.496Z" level=info msg="Created pod: artifact-passing-lfcp7[0].generate-artifact (artifact-passing-lfcp7-4182222514)" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.497Z" level=info msg="Workflow step group node artifact-passing-lfcp7-3140360656 not yet completed" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.497Z" level=info msg="TaskSet Reconciliation" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.497Z" level=info msg=reconcileAgentPod namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.497Z" level=info msg="Workflow to be dehydrated" Workflow Size=2042
time="2025-10-09T14:08:24Z" level=info msg="add pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:24Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:24.508Z" level=info msg="Workflow update successful" namespace=alida phase=Running resourceVersion=4270994 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.510Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=4270994 namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.511Z" level=info msg="Task-result reconciliation" namespace=alida numObjs=0 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.512Z" level=info msg="Workflow step group node artifact-passing-lfcp7-3140360656 not yet completed" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.512Z" level=info msg="TaskSet Reconciliation" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.512Z" level=info msg=reconcileAgentPod namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24.512Z" level=info msg="Workflow to be dehydrated" Workflow Size=2351
time="2025-10-09T14:08:24.518Z" level=info msg="Workflow update successful" namespace=alida phase=Running resourceVersion=4270997 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:24Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-4182222514
2025/10/09 14:08:24 http: TLS handshake error from 10.42.1.50:33148: client sent an HTTP request to an HTTPS server
time="2025-10-09T14:08:29Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:30Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:34.498Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=4270997 namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:34.499Z" level=info msg="Task-result reconciliation" namespace=alida numObjs=1 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:34.500Z" level=info msg="Workflow step group node artifact-passing-lfcp7-3140360656 not yet completed" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:34.500Z" level=info msg="TaskSet Reconciliation" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:34.500Z" level=info msg=reconcileAgentPod namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:34.500Z" level=info msg="Workflow to be dehydrated" Workflow Size=2406
time="2025-10-09T14:08:34.510Z" level=info msg="Workflow update successful" namespace=alida phase=Running resourceVersion=4271036 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:36Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:38Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:39Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:46.698Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=4271036 namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.698Z" level=info msg="Task-result reconciliation" namespace=alida numObjs=1 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.699Z" level=info msg="Step group node artifact-passing-lfcp7-3140360656 successful" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.699Z" level=info msg="node artifact-passing-lfcp7-3140360656 phase Running -> Succeeded" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.699Z" level=info msg="node artifact-passing-lfcp7-3140360656 finished: 2025-10-09 14:08:46.699876552 +0000 UTC" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.699Z" level=info msg="StepGroup node artifact-passing-lfcp7-2133556421 initialized Running" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.699Z" level=info msg="SG Outbound nodes of artifact-passing-lfcp7-4182222514 are [artifact-passing-lfcp7-4182222514]" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.700Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.700Z" level=info msg="Pod node artifact-passing-lfcp7-941708498 initialized Pending" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46Z" level=info msg="add pod event" pod=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:46.717Z" level=info msg="Created pod: artifact-passing-lfcp7[1].consume-artifact (artifact-passing-lfcp7-941708498)" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.717Z" level=info msg="Workflow step group node artifact-passing-lfcp7-2133556421 not yet completed" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.717Z" level=info msg="TaskSet Reconciliation" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.717Z" level=info msg=reconcileAgentPod namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46.717Z" level=info msg="Workflow to be dehydrated" Workflow Size=3443
time="2025-10-09T14:08:46Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:46.741Z" level=info msg="Workflow update successful" namespace=alida phase=Running resourceVersion=4271078 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:46Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=alida podName=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:46Z" level=info msg="cleaning up pod" action=labelPodCompleted key=alida/artifact-passing-lfcp7-4182222514/labelPodCompleted namespace=alida podName=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:46Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:46Z" level=info msg="delete pod event" pod=artifact-passing-lfcp7-4182222514
time="2025-10-09T14:08:47Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:48Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:51Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:53Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-941708498
2025/10/09 14:08:54 http: TLS handshake error from 10.42.1.50:37758: client sent an HTTP request to an HTTPS server
time="2025-10-09T14:08:54Z" level=info msg="update pod event" pod=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:56.717Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=4271078 namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.718Z" level=info msg="Task-result reconciliation" namespace=alida numObjs=2 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.718Z" level=info msg="SG Outbound nodes of artifact-passing-lfcp7-4182222514 are [artifact-passing-lfcp7-4182222514]" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="Step group node artifact-passing-lfcp7-2133556421 successful" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="node artifact-passing-lfcp7-2133556421 phase Running -> Succeeded" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="node artifact-passing-lfcp7-2133556421 finished: 2025-10-09 14:08:56.719144164 +0000 UTC" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="Outbound nodes of artifact-passing-lfcp7-941708498 is [artifact-passing-lfcp7-941708498]" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="Outbound nodes of artifact-passing-lfcp7 is [artifact-passing-lfcp7-941708498]" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="node artifact-passing-lfcp7 phase Running -> Succeeded" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="node artifact-passing-lfcp7 finished: 2025-10-09 14:08:56.719282874 +0000 UTC" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="TaskSet Reconciliation" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg=reconcileAgentPod namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="Updated phase Running -> Succeeded" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="Marking workflow completed" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="Marking workflow as pending archiving" namespace=alida workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56.719Z" level=info msg="Workflow to be dehydrated" Workflow Size=3854
time="2025-10-09T14:08:56.738Z" level=info msg="Workflow update successful" namespace=alida phase=Succeeded resourceVersion=4271132 workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56Z" level=info msg="queueing pod for cleanup" action=labelPodCompleted namespace=alida podName=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:56.759Z" level=info msg="archiving workflow" namespace=alida uid=949720a8-86ee-4d4c-b270-a06828c8536d workflow=artifact-passing-lfcp7
time="2025-10-09T14:08:56Z" level=info msg="cleaning up pod" action=labelPodCompleted key=alida/artifact-passing-lfcp7-941708498/labelPodCompleted namespace=alida podName=artifact-passing-lfcp7-941708498
time="2025-10-09T14:08:56Z" level=info msg="delete pod event" pod=artifact-passing-lfcp7-941708498

Logs from in your workflow's wait container

time="2025-10-09T14:08:48.519Z" level=info msg="Starting Workflow Executor" version=v3.7.0
time="2025-10-09T14:08:48.522Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-10-09T14:08:48.522Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=alida podName=artifact-passing-lfcp7-941708498 templateName= version="&Version{Version:v3.7.0,BuildDate:2025-07-23T08:51:02Z,GitCommit:d1d689b367c4c8a5a8a095ae60b5d7043f99eda9,GitTag:v3.7.0,GitTreeState:clean,GoVersion:go1.24.5,Compiler:gc,Platform:linux/amd64,}"
time="2025-10-09T14:08:48.611Z" level=info msg="Starting deadline monitor"
time="2025-10-09T14:08:52.613Z" level=info msg="Main container completed" error="<nil>"
time="2025-10-09T14:08:52.613Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-10-09T14:08:52.613Z" level=info msg="No output parameters"
time="2025-10-09T14:08:52.613Z" level=info msg="No output artifacts"
time="2025-10-09T14:08:52.614Z" level=info msg="S3 Save path: /tmp/argo/outputs/logs/main.log, key: artifact-passing-lfcp7/artifact-passing-lfcp7-941708498/main.log"
time="2025-10-09T14:08:52.614Z" level=info msg="Creating minio client using static credentials" endpoint="minio.minio:9000"
time="2025-10-09T14:08:52.614Z" level=info msg="Saving file to s3" bucket=argo-workflows endpoint="minio.minio:9000" key=artifact-passing-lfcp7/artifact-passing-lfcp7-941708498/main.log path=/tmp/argo/outputs/logs/main.log
time="2025-10-09T14:08:52.645Z" level=info msg="Save artifact" artifactName=main-logs duration=31.477597ms error="<nil>" key=artifact-passing-lfcp7/artifact-passing-lfcp7-941708498/main.log
time="2025-10-09T14:08:52.645Z" level=info msg="not deleting local artifact" localArtPath=/tmp/argo/outputs/logs/main.log
time="2025-10-09T14:08:52.645Z" level=info msg="Successfully saved file: /tmp/argo/outputs/logs/main.log"
time="2025-10-09T14:08:52.789Z" level=info msg="Alloc=10173 TotalAlloc=17358 Sys=22357 NumGC=4 Goroutines=9"
time="2025-10-09T14:08:52.878Z" level=info msg="Deadline monitor stopped"
time="2025-10-09T14:08:52.878Z" level=info msg="stopping progress monitor (context done)" error="context canceled"

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/gcGarbage collection, such as TTLs, retentionPolicy, delays, and more

    Type

    Bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions