Docker Hub tags not discovered for hours, then updater flaps between two latest tags (v0.12.x)

[dyefimov98]

Summary
Since 2025-09-11, argocd-image-updater stopped discovering newly pushed tags in several Docker Hub repositories. There were no error or warning logs. After a few hours, it started creating alternating commits that continuously switch the image tag back and forth between the two most recent tags (e.g., tag1 -> tag2 -> tag1 -> tag2 …). This is reproducible across three different clusters without any infrastructure changes. It only affects Docker Hub repositories that received a new tag on that day.

Expected behavior

New Docker Hub tags should be discovered promptly.
Updater should update once to the newest tag and remain stable (no oscillation).
Actual behavior

For several hours after new tags were pushed, updater did not detect them at all.
Later, updater began repeatedly flipping between the two latest tags, committing changes back and forth:
org/repo: tag1 -> tag2 -> tag1 -> tag2 … (multiple commits within minutes)
No errors or warnings at info level; debug logs show no obvious reason for the oscillation.
Environment

Kubernetes: DigitalOcean Kubernetes (DOKS) 1.31.1-do.4
argocd-image-updater Helm chart: 0.12.0 and 0.12.3 (same behavior on both)
argocd-image-updater image version: v0.16.0
Argo CD version: v2.14.3
Registry: Docker Hub
Authentication to Docker Hub: username-based
Timezone and approximate time window observed: UTC, 2025-09-11 19:00-now
Scope

Observed in 3 clusters, same configuration, same behavior.
Only Docker Hub repos affected, and only those that received new tags on 2025-09-11.
Steps to reproduce

Push a new tag to a Docker Hub repo used by argocd-image-updater (example: org/repo, push tag1, then tag2 shortly after).
Wait for argocd-image-updater to scan and update Git manifests.
Observe:
After some time, updater starts alternating commits switching between and .
Configuration snippets

App annotations:
Please see actual annotations we use:

  annotations:
    argocd-image-updater.argoproj.io/image-list: repo=dockerhuborg/{{image_name}}
    argocd-image-updater.argoproj.io/repo.update-strategy: newest-build
    argocd-image-updater.argoproj.io/repo.force-update: 'true'
    argocd-image-updater.argoproj.io/repo.allow-tags: regexp:^development-.*
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argo-cd/argocd-repo-github-repo
    argocd-image-updater.argoproj.io/git-branch: master
    argocd-image-updater.argoproj.io/repo.helm.image-name: image.repository
    argocd-image-updater.argoproj.io/repo.helm.image-tag: image.tag
    argocd-image-updater.argoproj.io/repo.pull-secret: pullsecret:argo-cd/dockerhub-secret

Logs

At info level: no errors or warnings when tags were not discovered.
At debug level: nothing obviously indicating failure or reason for alternating tag choice.

Additional context and observations

Issue began on 2025-09-11 without any config or infra changes on our side.
The oscillation only happens for repos that had new tags pushed that day.
Hypothesis: Docker Hub API may be returning unstable or equivalent metadata for the newest two tags (e.g., identical last_updated or same digest), and the updater’s tie-breaker causes non-deterministic selection, leading to flip-flopping commits.
We did not observe rate limit errors (429) or auth failures in logs.
Time skew should be minimal; NTP is enabled on nodes.

Workarounds tried

Switching update strategy (latest vs newest-build) did not resolve.
Upgrading from 0.12.0 to 0.12.3 did not change behavior.