failed to get message: failed to get next message: actions error: StatusCode 200, AcivityId "": EOF

[jmcgrathTT]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.12.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

N/A

Describe the bug

Our AutoscalingListener pod will stop accepting jobs and go into a "looping" state where we constantly see errors like this.

2025/10/23 16:04:46 Application returned an error: createSession failed: failed to create session: failed to do the session request: actions error: StatusCode 409, AcivityId "": GitHub.Actions.Runtime.WebApi.RunnerScaleSetSessionConflictException, GitHub.Actions.Runtime.WebApi: The actions runner scaleset example already has an active session.

The issue seems to be related to the 30-second cancel function context, as we don't ever see failed to delete message session: but only Deleting message session in the logs.

https://github.com/actions/actions-runner-controller/blob/gha-runner-scale-set-0.12.1/cmd/ghalistener/listener/listener.go#L441-L452

It seems like the message session is not being cleaned up in time, and when the app is re-initialized, it sees the old message session id via API causes it to "loop"

Describe the expected behavior

N/A

Additional Context

N/A

Controller Logs

2025/10/23 16:04:46 Application returned an error: createSession failed: failed to create session: failed to do the session request: actions error: StatusCode 409, AcivityId "": GitHub.Actions.Runtime.WebApi.RunnerScaleSetSessionConflictException, GitHub.Actions.Runtime.WebApi: The actions runner scaleset example already has an active session.
2025-10-23T16:02:07Z	INFO	listener-app	Starting listener
2025-10-23T16:02:07Z	INFO	listener-app	Starting metrics server
2025-10-23T16:02:07Z	INFO	listener-app	app initialized
2025/10/23 16:02:04 Application returned an error: failed to get message: failed to get next message: actions error: StatusCode 200, AcivityId "": EOF
2025-10-23T16:02:04Z	INFO	listener-app.listener	Deleting message session
2025-10-23T16:02:00Z	INFO	listener-app.listener	Getting next message	{"lastMessageID": 100042318}

Runner Pod Logs

N/A

[nikola-jokic]

Hey @jmcgrathTT,

Can you please help me understand how did you get into this situation?
If I understand correctly, your listener used to work, but exited for some reason. Then, the new one was created, having a session conflict? Do you have a log of the listener that used to work?

[jmcgrathTT]

Thanks for taking a look @nikola-jokic , I added more context.

If I understand correctly, your listener used to work, but exited for some reason. Then, the new one was created, having a session conflict?

We've been experiencing this for a few days ( since Tuesday, Oct 21st) with multiple gha-runner-scale-set. Not all of them have failed this way, but we have seen the RunnerScaleSetSessionConflictException occur 5 times in that timeframe for gha-runner-scale-set instances.

Do you have a log of the listener that used to work?

Logs are in the Controller Logs but I've added full logs to help.

I've seen two scenarios, but the listener controller's end state is always the same.

One is where the delete message session is called but fails it's function call.

1761093394109	2025-10-22T00:36:34.109Z	2025-10-22T00:36:34Z	INFO	listener-app.listener	Deleting message session

1761093394217	2025-10-22T00:36:34.217Z	2025-10-22T00:36:34Z	ERROR	listener-app.listener	failed to delete message session	{"error": "failed to delete message session: actions error: StatusCode 400, AcivityId \"\": GitHub.Actions.Runtime.WebApi.RunnerScaleSetSessionExpiredException, GitHub.Actions.Runtime.WebApi: The session identifier 4fd0ffb0-d365-4dae-99ff-8dae173984f9 is not valid."}

1761093394217	2025-10-22T00:36:34.217Z	github.com/actions/actions-runner-controller/cmd/ghalistener/listener.(*Listener).Listen.func1

1761093394217	2025-10-22T00:36:34.217Z		github.com/actions/actions-runner-controller/cmd/ghalistener/listener/listener.go:134

1761093394217	2025-10-22T00:36:34.217Z	github.com/actions/actions-runner-controller/cmd/ghalistener/listener.(*Listener).Listen

1761093394217	2025-10-22T00:36:34.217Z		github.com/actions/actions-runner-controller/cmd/ghalistener/listener/listener.go:165

1761093394217	2025-10-22T00:36:34.217Z	github.com/actions/actions-runner-controller/cmd/ghalistener/app.(*App).Run.func1

1761093394217	2025-10-22T00:36:34.217Z		github.com/actions/actions-runner-controller/cmd/ghalistener/app/app.go:130

1761093394217	2025-10-22T00:36:34.217Z	golang.org/x/sync/errgroup.(*Group).Go.func1

1761093394217	2025-10-22T00:36:34.217Z		golang.org/x/sync@v0.12.0/errgroup/errgroup.go:78

1761093394217	2025-10-22T00:36:34.217Z	2025-10-22T00:36:34Z	INFO	listener-app.metrics exporter.metrics	stopping metrics server	{"err": "context canceled"}

1761093394217	2025-10-22T00:36:34.217Z	2025/10/22 00:36:34 Application returned an error: failed to get message: refresh message session failed. failed to do the session request: actions error: StatusCode 400, AcivityId "": GitHub.Actions.Runtime.WebApi.RunnerScaleSetSessionExpiredException, GitHub.Actions.Runtime.WebApi: The session identifier 4fd0ffb0-d365-4dae-99ff-8dae173984f9 is not valid.

From here, the app will re-initialize (no container restart) and loop this log.

1761093403090	2025-10-22T00:36:43.090Z	2025/10/22 00:36:43 Application returned an error: createSession failed: failed to create session: failed to do the session request: actions error: StatusCode 409, AcivityId "": GitHub.Actions.Runtime.WebApi.RunnerScaleSetSessionConflictException, GitHub.Actions.Runtime.WebApi: The actions runner scaleset example-label already has an active session.

Full logs: listener-logs-failed-func-message-session-delete.txt

The second one is the Deleting message session message, but you never see the failed function call. Seems to be related to the 30 second context here https://github.com/actions/actions-runner-controller/blob/gha-runner-scale-set-0.12.1/cmd/ghalistener/listener/listener.go#L441-L452

1761235324127	2025-10-23T16:02:04.127Z	2025-10-23T16:02:04Z	INFO	listener-app.listener	Deleting message session

1761235324275	2025-10-23T16:02:04.275Z	2025-10-23T16:02:04Z	INFO	listener-app.metrics exporter.metrics	stopping metrics server	{"err": "context canceled"}

1761235324275	2025-10-23T16:02:04.275Z	2025/10/23 16:02:04 Application returned an error: failed to get message: failed to get next message: actions error: StatusCode 200, AcivityId "": EOF

And just as before, the app will re-initialize (no container restart) and loop this log.

1761093403090	2025-10-22T00:36:43.090Z	2025/10/22 00:36:43 Application returned an error: createSession failed: failed to create session: failed to do the session request: actions error: StatusCode 409, AcivityId "": GitHub.Actions.Runtime.WebApi.RunnerScaleSetSessionConflictException, GitHub.Actions.Runtime.WebApi: The actions runner scaleset example-label already has an active session.

Full logs: listener-logs-delete-message-session-func-cancelled.txt

Regardless of how it happens, the listener always ends up in this state. The fix has been to delete the Helm deployment and re-deploy the Helm chart. To me, it seems that the ActivityId is getting lost when we re-initialize the app (in memory), and only a redeploy fixes it.