High CPU usage

[0x6D682D7331]

Hi,
I encountered high CPU usage with dry on small Docker Swarm cluster (2 nodes).

Without dry:

With dry:

When i killed dry, journal showed a lot of entries like:

Mar 13 13:38:57  01-app-01-master dockerd[970]: 2025/03/13 13:38:57 http: superfluous response.WriteHeader call from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).WriteHeader (wra>
Mar 13 13:38:57  01-app-01-master dockerd[970]: time="2025-03-13T13:38:57.942650324+01:00" level=error msg="Handler for GET /v1.44/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pip>
Mar 13 13:38:57  01-app-01-master dockerd[970]: 2025/03/13 13:38:57 http: superfluous response.WriteHeader call from go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*respWriterWrapper).WriteHeader (wra>
Mar 13 13:38:57  01-app-01-master dockerd[970]: time="2025-03-13T13:38:57.940368419+01:00" level=error msg="Handler for GET /v1.44/containers/json returned error: write unix /var/run/docker.sock->@: write: broken pip>

Dry version:

dry version 0.11.2, build fe820a70

Docker info:

Client: Docker Engine - Community
 Version:    27.5.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 128
  Running: 54
  Paused: 0
  Stopped: 74
 Images: 119
 Server Version: 27.5.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: gagjjfzq8t812yd0hkmd9zwaf
  Is Manager: true
  ClusterID: enkheij9i5vq273wmgyk6i13s
  Managers: 1
  Nodes: 2
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 1
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.234.12.67
  Manager Addresses:
   10.234.12.67:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-30-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.34GiB
 Name: app-01-master
 ID: 9ca1d0fc-e68c-4204-98ef-9065947c3dce
 Docker Root Dir: /mnt/why/not/secure
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker node:

 docker node ls
ID                            HOSTNAME                      STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
gagjjfzq8t812yd0hkmd9zwaf *   app-01-master   Ready     Active         Leader           27.5.1
q47f0ez3jbnvarvthh43jhtgy     app-02-master   Ready     Active                          27.5.1

[damntourists]

Also facing this issue on Oracle Linux 9 (kernel-uek-6.12.0-1.23.3.2). Seems as soon as this line is hit https://github.com/moncho/dry/blob/master/app/dry.go#L147, the system begins to fill up all available ram and shortly after fills all available swap. Tried this on docker versions 27.5.1 and 28.2.2. A similar report was filed at moby/moby#50132, though I was unable to correct the issues using any methods described using the latest pull from this repository. Currently the go.mod file shows that docker v27.5.1+incompatible is in use. Perhaps it would be corrected if updated to the latest version?

[moncho]

Good finding, @damntourists! Updating the dependencies breaks a few things, so it is not a small change. Anyway, good to know that this problem will be fixed with a new release.

[damntourists]

I wanted to follow up on this. This seems to still be happening with 28.2.2. This part of the code https://github.com/moncho/dry/blob/master/docker/daemon.go#L539 will trigger the memory leak when Size: true, however the memory leak does not occur when Size: false.