RSDK-11788 Forward recent systemd agent logs upon startup #184
+253
−57
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benjirewis
commented
Jan 14, 2026
Member
Author
benjirewis
left a comment
benjirewis
left a comment
There was a problem hiding this comment.
This PR is still in "draft" since I'm adding tests and want to make sure "OOM killed" logs specifically get saved, but feel free to review @danielbotros + @jmatth .
benjirewis
force-pushed
the
recent-systemd-logs-forwarding
branch
from
103c870 to
6759c78
Compare
Member
Author
|
Tests added; now ready for full review @jmatth + @danielbotros . |
jmatth
reviewed
Jan 23, 2026
Comment on lines
+39
to
+45
| // A nil lastShutdown value means none was saved in the cache. Bail in this case. We | ||
| // _could_ attempt to upload all historical systemd agent logs if we do not know when | ||
| // the agent last shutdown, but there may be quite a few of these logs, and doing so | ||
| // could slow startup. | ||
| if s.lastShutdown == nil { | ||
| return nil | ||
| } |
Member
There was a problem hiding this comment.
I think there are some edge cases here:
- If agent gets OOM killed or similar during its first run we won't upload any logs on the next start
- If agent gets OOM killed or similar after one or more clean shutdowns then we're going to upload all the systemd logs since the last shutdown. I'm assuming agent has average uptimes in days or weeks so that could still be a lot of logs. Is it the case that
viam-agent.systemdlogs are only created very rarely during the lifetime of the process?
| journalCmd *exec.Cmd | ||
| cancelFunc context.CancelFunc | ||
| noJournald bool | ||
| lastShutdown *time.Time |
Member
There was a problem hiding this comment.
Needing to keep this extra value on the struct even though it's only really used during startup is kind of annoying. I don't see a good way around it given the current code though. Just flagging it as one more reason we should get rid of the weird subsystem duck typing at some point.
| } | ||
|
|
||
| func TestVersionCacheJSONRoundtrip(t *testing.T) { | ||
| someTime, err := time.Parse("2006-01-02 15:04:05", "2011-11-11 00:00:00" /* https://tinyurl.com/dm4ytr3c */) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
RSDK-11788
Adds a
last_shutdowntimestamp to the version cache. Upon startup, usesjournalctl -u viam-agent -t systemd -o json -S [last_shutdown]to forward allsystemd-identified logs from theviam-agentsystemd service since the last shutdown to app. This allows logs like the following to make it up to app: