Skip to content

Nil pointer dereference due to malformed CronWorkflow. Workflow-controller crashlooping #14968

@bjoern-weidlich-anchorage

Description

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Controller version: argoproj/workflow-controller:v3.7.2

Current Behavior:

The workflow-controller panics and enters a crash loop when it attempts to process a CronWorkflow that is malformed (likely with a missing or nil field within its workflowSpec).

Instead of marking the CronWorkflow as invalid and logging an error, the controller panics with runtime error: "invalid memory address or nil pointer dereference." and enters a crashloop.

Expected Behavior:
The controller should gracefully handle malformed CronWorkflow resources. It should:

  • Fail to validate the resource.
  • Update the .status of the CronWorkflow with a condition indicating the validation error.
  • Log a clear error message (like "Failed to validate CronWorkflow...").

Continue running without crashing to manage other valid resources.

Logs from the workflow controller:

Status object from the problematic CronWorkflow resource:

Status:                                                                                                                                                                                           
  Conditions:                                                                                                                                                                                     
    Message:  cannot validate Workflow: template name 'redacted' undefined                                                                                                                
    Status:   True                                                                                                                                                                                
    Type:     SpecError                                                                                                                                                                           
  Failed:     0                                                                                                                                                                                   
  Phase:                                                                                                                                                                                          
  Succeeded:  0      

Version(s)

v3.7.2

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

Create a Cronworkflow with a missing template?

Logs from the workflow controller

time="2025-10-22T21:42:09.319Z" level=info msg="Processing redacted/redacted" cronWorkflow=redacted/redacted
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x1ee75c1]

goroutine 236 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x343f9f0, 0xc0000120e0}, {0x2aa4980, 0x4e1bde0}, {0x0, 0x0, 0x440d80?})
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:114 +0x1a9
k8s.io/apimachinery/pkg/util/runtime.HandleCrashWithContext({0x343f9f0, 0xc0000120e0}, {0x0, 0x0, 0x0})
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:78 +0x5a
panic({0x2aa4980?, 0x4e1bde0?})
    /usr/local/go/src/runtime/panic.go:792 +0x132
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x343f138, 0x4eab0e0}, {0x2aa4980, 0x4e1bde0}, {0x4e19990, 0x1, 0xc0005bf300?})
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:114 +0x1a9
k8s.io/apimachinery/pkg/util/runtime.HandleCrashWithContext({0x343f138, 0x4eab0e0}, {0x4e19990, 0x1, 0x1})
    /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:78 +0x5a
panic({0x2aa4980?, 0x4e1bde0?})
    /usr/local/go/src/runtime/panic.go:792 +0x132
[github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).addPodMetadata](https://github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).addPodMetadata)(...)
    /go/src/[github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:295](https://github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:295)
[github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).GetTemplateByName(0xc001e5d0e8](https://github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).GetTemplateByName(0xc001e5d0e8), {0xc0014d4c90, 0x10})
    /go/src/[github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:122](https://github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:122) +0xe1
[github.com/argoproj/argo-workflows/v3/workflow/validate.(*templateValidationCtx).validateTemplateHolder(0xc001ef2220](https://github.com/argoproj/argo-workflows/v3/workflow/validate.(*templateValidationCtx).validateTemplateHolder(0xc001ef2220), {0x3446620, 0xc0020d63c0}, 0xc001e5d0e8, {0x3413088, 0xc002ca0d50}, 0x0)
    /go/src/[github.com/argoproj/argo-workflows/workflow/validate/validate.go:565](https://github.com/argoproj/argo-workflows/workflow/validate/validate.go:565) +0xab
[github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateWorkflow](https://github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateWorkflow)({0x34037e0, 0xc001ef2180}, {0x3403800, 0xc002e037b0}, 0xc002f02488, 0x0, {0x0, 0x0, 0x0, 0x0})
    /go/src/[github.com/argoproj/argo-workflows/workflow/validate/validate.go:257](https://github.com/argoproj/argo-workflows/workflow/validate/validate.go:257) +0x219e
[github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateCronWorkflow](https://github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateCronWorkflow)({0x343f058?, 0xc002ca0ba0?}, {0x34037e0, 0xc001ef2180}, {0x3403800, 0xc002e037b0}, 0xc00227c408, 0x0)
    /go/src/[github.com/argoproj/argo-workflows/workflow/validate/validate.go:404](https://github.com/argoproj/argo-workflows/workflow/validate/validate.go:404) +0x34d
[github.com/argoproj/argo-workflows/v3/workflow/cron.(*cronWfOperationCtx).validateCronWorkflow(0xc000630630](https://github.com/argoproj/argo-workflows/v3/workflow/cron.(*cronWfOperationCtx).validateCronWorkflow(0xc000630630), {0x343f058, 0xc002ca0ba0})
    /go/src/[github.com/argoproj/argo-workflows/workflow/cron/operator.go:146](https://github.com/argoproj/argo-workflows/workflow/cron/operator.go:146) +0x15f
[github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).processNextCronItem(0xc000246460](https://github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).processNextCronItem(0xc000246460), {0x343f138, 0x4eab0e0})
    /go/src/[github.com/argoproj/argo-workflows/workflow/cron/controller.go:182](https://github.com/argoproj/argo-workflows/workflow/cron/controller.go:182) +0x8c5
[github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).runCronWorker](https://github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).runCronWorker)(...)
    /go/src/[github.com/argoproj/argo-workflows/workflow/cron/controller.go:136](https://github.com/argoproj/argo-workflows/workflow/cron/controller.go:136)

Logs from in your workflow's wait container

does not have a wait container?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions