High CPU & RAM Usage on ARM32 and ARM64

[bella-WI]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Ory Network Project

No response

Describe the bug

Hey guys,
we are using hydra in our embedded devices as an oauth2-server for machine client access. So far everything worked like a charm and we didn't have any issues.

After updating to version 2.3.0 we are facing issues with high cpu und ram loads. This only happens as soon as the time of the system is increased to a date in the future. For example I set the timestamp to the 10.10.2027. And resource consumption increases.
When I set the timestamp even more in the future, the resources are exhausted for a longer time, so that the kernel kills the process. It seems like some cleanup is happening during that time.
Could this be related to these lines?

hydra/client/handler.go

Line 828 in ee8c339

fosite.AccessToken: time.Now().Add(time.Hour),

Is this a known issue? In 2.2.0 this issue didn't happen.
My tests could not pinpoint the cause.

Reproducing the bug

Steps to reproduce.

1. Start hydra on arm32 arm64 architecture
2. increase the timestamp to a date in the future
3. see cpu and ram sky rocket

Relevant log output

Unfortunately no logs from hydra

Relevant configuration

Version

2.3.0

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Binary

Additional Context

No response

[fnoehWM]

Root Cause: Ristretto Cache Cleanup on Time Jump

Following up on the investigation - I've identified the root cause via CPU profiling and source code analysis.

CPU Profile Evidence

After a 10-year time jump forward:

• 94.73% CPU usage (vs 5.39% normal)
• 52% of CPU time spent in github.com/dgraph-io/ristretto.(*expirationMap[...]).cleanup
• Heavy GC pressure from processing expired entries

The Issue

Ristretto's TTL implementation uses 5-second time buckets (ttl.go). The cleanup() method iterates through all time buckets between lastCleanedBucketNum and the current time.

On a 10-year time jump: currentBucketNum - lastCleanedBucketNum ≈ 63 million iterations, causing the CPU spike.

Affected Component

From the CPU profile, at least one Ristretto cache using github.com/ory/x/jsonschemax.Path types is affected, specifically the cache created in ory/x/configx/schema_path_cache.go. This is probably not the only cache affected - ory/x v0.0.675 uses Ristretto in several other modules as well.

When NewCache() is called, it creates a cleanupTicker that schedules regular cleanup operations. The timer itself is likely not the cause - it's just a scheduler. The actual issue is the unbounded bucket iteration in the cleanup() logic described in "The Issue" section above, which runs when the timer fires.

Version Context

• Issue present in v2.3.0 (ory/x v0.0.675)
• Not present in v2.2.0 (ory/x v0.0.612)

Impact

Systems experiencing time jumps (NTP sync on embedded devices, VM pause/resume, manual time changes) see sustained high CPU usage until cleanup completes.

Potential Fix

A straightforward fix might be to calculate the number of bucket iterations in the cleanup loop with the timer library, instead of Date calculation. Or just limit them to e.g. 1000. However, we don't have sufficient insight into how this would impact other components using Ristretto.

Happy to provide additional CPU profiles or if needed. A profile summary is included below:

---

CPU Profile Analysis (Click to expand)

Created with: go tool pprof -text cpu.pprof > cpu-analysis.txt

File: hydra
Build ID: eac1ad78c337a677dd682e0504421d4e5875a76b
Type: cpu
Time: 2025-10-16 14:57:58 CEST
Duration: 26.60s, Total samples = 25.20s (94.73%)
Showing nodes accounting for 24.16s, 95.87% of 25.20s total
Dropped 187 nodes (cum <= 0.13s)
flat flat% sum% cum cum%
3.96s 15.71% 15.71% 5.29s 20.99% runtime.typePointers.next
3.50s 13.89% 29.60% 8.82s 35.00% runtime.bulkBarrierPreWriteSrcOnly
3.36s 13.33% 42.94% 3.36s 13.33% runtime.tgkill
2.35s 9.33% 52.26% 2.35s 9.33% runtime.memmove
1.67s 6.63% 58.89% 1.67s 6.63% runtime.osyield
1.43s 5.67% 64.56% 1.43s 5.67% runtime.nanotime (inline)
1.15s 4.56% 69.13% 1.71s 6.79% runtime.(*wbBuf).get1 (inline)
1.10s 4.37% 73.49% 8.66s 34.37% runtime.suspendG
0.81s 3.21% 76.71% 0.81s 3.21% runtime.typePointers.nextFast (inline)
0.76s 3.02% 79.72% 2.63s 10.44% runtime.scanobject
0.57s 2.26% 81.98% 0.57s 2.26% runtime.procyield
0.56s 2.22% 84.21% 0.56s 2.22% runtime.readUintptr (inline)
0.56s 2.22% 86.43% 0.59s 2.34% runtime.wbBufFlush1
0.45s 1.79% 88.21% 0.45s 1.79% runtime.nanotime1
0.37s 1.47% 89.68% 5.54s 21.98% github.com/dgraph-io/ristretto.(*expirationMap[go.shape.*uint8]).cleanup
0.34s 1.35% 91.03% 7.61s 30.20% github.com/dgraph-io/ristretto.(*expirationMap[go.shape.[]github.com/ory/x/jsonschemax.Path]).cleanup
0.31s 1.23% 92.26% 0.31s 1.23% runtime.getpid
0.25s 0.99% 93.25% 0.34s 1.35% gcWriteBarrier
0.22s 0.87% 94.13% 0.22s 0.87% runtime.mapaccess1_fast64
0.20s 0.79% 94.92% 0.20s 0.79% runtime.mapdelete_fast64
0.07s 0.28% 95.20% 0.22s 0.87% github.com/ory/x/popx.Migrations.Less
0.05s 0.2% 95.40% 0.22s 0.87% sort.partition
0.03s 0.12% 95.52% 3.70s 14.68% runtime.signalM
0.02s 0.079% 95.60% 10.85s 43.06% runtime.gcDrain
0.02s 0.079% 95.67% 12.14s 48.17% runtime.systemstack
0.02s 0.079% 95.75% 0.61s 2.42% runtime.wbBufFlush.func1
0.01s 0.04% 95.79% 3.73s 14.80% runtime.preemptM (inline)
0.01s 0.04% 95.83% 0.65s 2.58% runtime.wbBufFlush
0.01s 0.04% 95.87% 0.13s 0.52% sort.insertionSort
0 0% 95.87% 5.54s 21.98% github.com/dgraph-io/ristretto.(*Cache[go.shape.[]uint8,go.shape.*uint8]).processItems
0 0% 95.87% 7.61s 30.20% github.com/dgraph-io/ristretto.(*Cache[go.shape.[]uint8,go.shape.[]github.com/ory/x/jsonschemax.Path]).processItems
0 0% 95.87% 5.54s 21.98% github.com/dgraph-io/ristretto.(*shardedMap[go.shape.*uint8]).Cleanup
0 0% 95.87% 7.61s 30.20% github.com/dgraph-io/ristretto.(*shardedMap[go.shape.[]github.com/ory/x/jsonschemax.Path]).Cleanup
0 0% 95.87% 0.48s 1.90% github.com/ory/hydra/v2/cmd.Execute
0 0% 95.87% 0.48s 1.90% github.com/ory/hydra/v2/cmd.RegisterCommandRecursive.NewServeAllCmd.RunServeAll.func12
0 0% 95.87% 0.43s 1.71% github.com/ory/hydra/v2/driver.(*RegistrySQL).Init
0 0% 95.87% 0.48s 1.90% github.com/ory/hydra/v2/driver.New
0 0% 95.87% 0.42s 1.67% github.com/ory/hydra/v2/persistence/sql.NewPersister
0 0% 95.87% 0.41s 1.63% github.com/ory/x/popx.(*MigrationBox).findMigrations (inline)
0 0% 95.87% 0.42s 1.67% github.com/ory/x/popx.NewMigrationBox
0 0% 95.87% 0.40s 1.59% github.com/ory/x/popx.NewMigrationBox.(*MigrationBox).findMigrations.func4
0 0% 95.87% 0.48s 1.90% github.com/spf13/cobra.(*Command).Execute (inline)
0 0% 95.87% 0.48s 1.90% github.com/spf13/cobra.(*Command).ExecuteC
0 0% 95.87% 0.48s 1.90% github.com/spf13/cobra.(*Command).execute
0 0% 95.87% 0.41s 1.63% io/fs.WalkDir
0 0% 95.87% 0.41s 1.63% io/fs.walkDir
0 0% 95.87% 0.48s 1.90% main.main
0 0% 95.87% 0.50s 1.98% runtime.deductAssistCredit
0 0% 95.87% 0.50s 1.98% runtime.gcAssistAlloc
0 0% 95.87% 0.49s 1.94% runtime.gcAssistAlloc.func1
0 0% 95.87% 0.49s 1.94% runtime.gcAssistAlloc1
0 0% 95.87% 10.97s 43.53% runtime.gcBgMarkWorker
0 0% 95.87% 10.85s 43.06% runtime.gcBgMarkWorker.func2
0 0% 95.87% 5.99s 23.77% runtime.gcDrainMarkWorkerDedicated (inline)
0 0% 95.87% 4.86s 19.29% runtime.gcDrainMarkWorkerIdle (inline)
0 0% 95.87% 0.49s 1.94% runtime.gcDrainN
0 0% 95.87% 11.68s 46.35% runtime.growslice
0 0% 95.87% 0.48s 1.90% runtime.main
0 0% 95.87% 0.58s 2.30% runtime.mallocgc
0 0% 95.87% 8.69s 34.48% runtime.markroot
0 0% 95.87% 8.66s 34.37% runtime.markroot.func1
0 0% 95.87% 0.36s 1.43% sort.Sort
0 0% 95.87% 0.36s 1.43% sort.pdqsort

[fnoehWM]

Update: Already Fixed in ory/x

This was already fixed upstream in ristretto#429 (merged Dec 31, 2024) and ory/x upgraded to Ristretto v2.1.0 in commit fb8c4a3 (March 12, 2025).

Current status:

• ✅ Ristretto v2.1.0+ has the fix
• ✅ ory/x master upgraded to v2.1.0
• ⏳ Hydra v2.3.0 still uses ory/x v0.0.675 (with Ristretto v1.0.0)

The issue will be resolved in the next Hydra release that bumps the ory/x dependency.