-
Notifications
You must be signed in to change notification settings - Fork 3.4k
Closed
Bug
Copy link
Description
Pre-requisites
- I have double-checked my configuration
- I have tested with the
:latestimage tag (i.e.quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on:latest. If not, I have explained why, in detail, in my description below. - I have searched existing issues and could not find a match for this bug
- I'd like to contribute the fix myself (see contributing guide)
What happened? What did you expect to happen?
Controller version: argoproj/workflow-controller:v3.7.2
Current Behavior:
The workflow-controller panics and enters a crash loop when it attempts to process a CronWorkflow that is malformed (likely with a missing or nil field within its workflowSpec).
Instead of marking the CronWorkflow as invalid and logging an error, the controller panics with runtime error: "invalid memory address or nil pointer dereference." and enters a crashloop.
Expected Behavior:
The controller should gracefully handle malformed CronWorkflow resources. It should:
- Fail to validate the resource.
- Update the .status of the CronWorkflow with a condition indicating the validation error.
- Log a clear error message (like "Failed to validate CronWorkflow...").
Continue running without crashing to manage other valid resources.
Logs from the workflow controller:
Status object from the problematic CronWorkflow resource:
Status:
Conditions:
Message: cannot validate Workflow: template name 'redacted' undefined
Status: True
Type: SpecError
Failed: 0
Phase:
Succeeded: 0
Version(s)
v3.7.2
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
Create a Cronworkflow with a missing template?Logs from the workflow controller
time="2025-10-22T21:42:09.319Z" level=info msg="Processing redacted/redacted" cronWorkflow=redacted/redacted
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x1ee75c1]
goroutine 236 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x343f9f0, 0xc0000120e0}, {0x2aa4980, 0x4e1bde0}, {0x0, 0x0, 0x440d80?})
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:114 +0x1a9
k8s.io/apimachinery/pkg/util/runtime.HandleCrashWithContext({0x343f9f0, 0xc0000120e0}, {0x0, 0x0, 0x0})
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:78 +0x5a
panic({0x2aa4980?, 0x4e1bde0?})
/usr/local/go/src/runtime/panic.go:792 +0x132
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x343f138, 0x4eab0e0}, {0x2aa4980, 0x4e1bde0}, {0x4e19990, 0x1, 0xc0005bf300?})
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:114 +0x1a9
k8s.io/apimachinery/pkg/util/runtime.HandleCrashWithContext({0x343f138, 0x4eab0e0}, {0x4e19990, 0x1, 0x1})
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:78 +0x5a
panic({0x2aa4980?, 0x4e1bde0?})
/usr/local/go/src/runtime/panic.go:792 +0x132
[github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).addPodMetadata](https://github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).addPodMetadata)(...)
/go/src/[github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:295](https://github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:295)
[github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).GetTemplateByName(0xc001e5d0e8](https://github.com/argoproj/argo-workflows/v3/workflow/templateresolution.(*Context).GetTemplateByName(0xc001e5d0e8), {0xc0014d4c90, 0x10})
/go/src/[github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:122](https://github.com/argoproj/argo-workflows/workflow/templateresolution/context.go:122) +0xe1
[github.com/argoproj/argo-workflows/v3/workflow/validate.(*templateValidationCtx).validateTemplateHolder(0xc001ef2220](https://github.com/argoproj/argo-workflows/v3/workflow/validate.(*templateValidationCtx).validateTemplateHolder(0xc001ef2220), {0x3446620, 0xc0020d63c0}, 0xc001e5d0e8, {0x3413088, 0xc002ca0d50}, 0x0)
/go/src/[github.com/argoproj/argo-workflows/workflow/validate/validate.go:565](https://github.com/argoproj/argo-workflows/workflow/validate/validate.go:565) +0xab
[github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateWorkflow](https://github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateWorkflow)({0x34037e0, 0xc001ef2180}, {0x3403800, 0xc002e037b0}, 0xc002f02488, 0x0, {0x0, 0x0, 0x0, 0x0})
/go/src/[github.com/argoproj/argo-workflows/workflow/validate/validate.go:257](https://github.com/argoproj/argo-workflows/workflow/validate/validate.go:257) +0x219e
[github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateCronWorkflow](https://github.com/argoproj/argo-workflows/v3/workflow/validate.ValidateCronWorkflow)({0x343f058?, 0xc002ca0ba0?}, {0x34037e0, 0xc001ef2180}, {0x3403800, 0xc002e037b0}, 0xc00227c408, 0x0)
/go/src/[github.com/argoproj/argo-workflows/workflow/validate/validate.go:404](https://github.com/argoproj/argo-workflows/workflow/validate/validate.go:404) +0x34d
[github.com/argoproj/argo-workflows/v3/workflow/cron.(*cronWfOperationCtx).validateCronWorkflow(0xc000630630](https://github.com/argoproj/argo-workflows/v3/workflow/cron.(*cronWfOperationCtx).validateCronWorkflow(0xc000630630), {0x343f058, 0xc002ca0ba0})
/go/src/[github.com/argoproj/argo-workflows/workflow/cron/operator.go:146](https://github.com/argoproj/argo-workflows/workflow/cron/operator.go:146) +0x15f
[github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).processNextCronItem(0xc000246460](https://github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).processNextCronItem(0xc000246460), {0x343f138, 0x4eab0e0})
/go/src/[github.com/argoproj/argo-workflows/workflow/cron/controller.go:182](https://github.com/argoproj/argo-workflows/workflow/cron/controller.go:182) +0x8c5
[github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).runCronWorker](https://github.com/argoproj/argo-workflows/v3/workflow/cron.(*Controller).runCronWorker)(...)
/go/src/[github.com/argoproj/argo-workflows/workflow/cron/controller.go:136](https://github.com/argoproj/argo-workflows/workflow/cron/controller.go:136)
Logs from in your workflow's wait container
does not have a wait container?