master日志一直写入，快速达到设置的上限

[liangkiller]

版本3.0，3个节点的集群。 master的日志一直不断的写入，快速达到默认的上限50G。 日志一直输出以下内容，麻烦看下是什么原因

[ERROR] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[688] - [WorkflowInstance-11][TaskInstance-0] - Start workflow error
java.lang.NullPointerException: null
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowEventLooper:[79] - [WorkflowInstance-11][TaskInstance-0] - Workflow event looper receive
 a workflow event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11), will handle this
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowEventQueue:[38] - [WorkflowInstance-0][TaskInstance-0] - Added workflow event to workflowEvent queue, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11)
[ERROR] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[74] - [WorkflowInstance-0][TaskInstance-0] - Failed to submit the workflow instance, will resend the workflow start event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11)
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[54] - [WorkflowInstance-11][TaskInstance-0] - Handle workflow start event, begin to start a workflow, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11)
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowEventQueue:[38] - [WorkflowInstance-0][TaskInstance-0] - Added workflow event to workflowEvent queue, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11)
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowEventLooper:[79] - [WorkflowInstance-11][TaskInstance-0] - Workflow event looper receive a workflow event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11), will handle this
[ERROR] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[688] - [WorkflowInstance-11][TaskInstance-0] - Start workflow error
java.lang.NullPointerException: null
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[54] - [WorkflowInstance-11][TaskInstance-0] - Handle workflow start event, begin to start a workflow, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11)
[ERROR] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowStartEventHandler:[74] - [WorkflowInstance-0][TaskInstance-0] - Failed to submit the workflow instance, will resend the workflow start event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11)
[ERROR] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[688] - [WorkflowInstance-11][TaskInstance-0] - Start workflow error
java.lang.NullPointerException: null
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowEventLooper:[79] - [WorkflowInstance-11][TaskInstance-0] - Workflow event looper receive a workflow event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11), will handle this
[INFO] 2022-09-14 11:06:18.493 +0800 org.apache.dolphinscheduler.server.master.event.WorkflowEventQueue:[38] - [WorkflowInstance-0][TaskInstance-0] - Added workflow event to workflowEvent queue, event: WorkflowEvent(workflowEventType=START_WORKFLOW, workflowInstanceId=11)

任务都是能正常运行的

[liangkiller]

是shell类型的任务， 看了工作流实例11，状态是 ：失败， 运行类型是：启动工作流； 相应的任务实例 是失败。

[dandan0203]

一般造成这种问题的原因是你任务刚执行的时候，你按了停止按钮，解决的话一般到mysql数据库找到该任务实例，删除他，并重启海豚

[inventertom]

我遇到 同样的情况，4个节点，2个master,其中一台master 突然down 机，重启后，master挂掉的master, 这个节点的matser 日志一直在写入，快速打满
日志内容如下：
[ERROR] 2023-08-06 12:08:17.698 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[278] - [WorkflowInstance-4033140][TaskInstance-9392560] - State event handle error, will retry this event: StateEvent(key=4033142-0-4033140-9392560, type=TA
SK_STATE_CHANGE, executionStatus=RUNNING_EXECUTION, taskInstanceId=9392560, taskCode=0, processInstanceId=4033140, context=null, channel=null)
org.apache.dolphinscheduler.server.master.event.StateEventHandleException: Task state event handle error, due to the task is not in activeTaskProcessorMaps
at org.apache.dolphinscheduler.server.master.event.TaskStateEventHandler.handleStateEvent(TaskStateEventHandler.java:84)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable.handleEvents(WorkflowExecuteRunnable.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[ERROR] 2023-08-06 12:08:17.698 +0800 org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable:[278] - [WorkflowInstance-4043260][TaskInstance-9495666] - State event handle error, will retry this event: StateEvent(key=4043261-0-4043260-9495666, type=TA
SK_STATE_CHANGE, executionStatus=RUNNING_EXECUTION, taskInstanceId=9495666, taskCode=0, processInstanceId=4043260, context=null, channel=null)
org.apache.dolphinscheduler.server.master.event.StateEventHandleException: Task state event handle error, due to the task is not in activeTaskProcessorMaps
at org.apache.dolphinscheduler.server.master.event.TaskStateEventHandler.handleStateEvent(TaskStateEventHandler.java:84)
at org.apache.dolphinscheduler.server.master.runner.WorkflowExecuteRunnable.handleEvents(WorkflowExecuteRunnable.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java

做过如下尝试：
1、根据taskInstance 修改任务实例的状态 , 重启此节点的master ,现象依旧
2、根据taskInstance 删除任务实例，重启此节点master ,现象依旧

请问有什么其他解决办法？

[zhujianjun692-wq]

我也遇到同样问题，做过如下尝试：
1.根据taskInstance 修改任务实例的状态，置成功或失败，日志还是一直打印，且该作业还是无法正常下发给worker节点执行；
2.根据taskInstance 删除任务实例，重启dolphinscheduler 集群，日志还是一直打印，报错，页面显示该任务出现容错，mysql修改该任务容错状态，改为成功，再删除，还是无法解决日志增长问题；