[Bug] [master] Task dispatch queue blocking problem

[buddhist-coder]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

When a Worker node goes offline, all tasks stop executing (not just the tasks assigned to the offline Worker). Even if tasks are assigned to other normally running Workers, they will no longer be executed.

What you expected to happen

• The dispatchFailTimes of failed tasks will increase.

• In theory, tasks with frequent failures should have their priority lowered.

• However, if other priority attributes of the failed task (such as workflow priority or task priority) are higher, it will still be placed at the front of the queue.

• This will cause each take() call to retrieve the same failed task, preventing other tasks from ever being executed.

How to reproduce

• There are several tasks in the queue.

• One of these tasks requires a non-existent Worker Group and continues to fail.

• This task is removed from the queue each time it fails and then re-enqueued.

• Other normal tasks never get a chance to execute.

• All tasks are in the "Submit Successful" state, but are actually suspended.

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[SbloodyS]

This issue should be fix in 3.3.X. Please try the latest version.

[buddhist-coder]

I found the bug in version 3.2.2

[ruanwenjun]

This should already been fixed at 3.3.x