Aufs to overlay2 migration failure

[alexgg]

Description

After a hostOS update on a balenaFin that performs an aufs to overlay migration, the engine in the newOS fails to perform the migration and errors with:

Jun 23 14:40:18 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:18.955898073Z" level=info msg="Storage migration from aufs to overlay2 starting"
Jun 23 14:40:45 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:45.998202208Z" level=info msg="transforming 342 layers(s) to overlay2"
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.001704023Z" level=info msg="Storage migration finished, took 27.045128503s"
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.001925741Z" level=error msg="failed aufs to overlay2 migration" error="Error moving layer data to overlay2: symlink hostname /var/lib/docker/overlay2.temp/00d0a01127ca0>
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.002066418Z" level=warning msg="recovering from failed aufs to overlay migration" storage_root=/var/lib/docker
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.237646393Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
Jun 23 14:40:46 a5fa2e0 balenad[1995]: time="2022-06-23T14:40:46.237842799Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
Jun 23 14:40:47 a5fa2e0 balenad[1995]: failed to start daemon: error cleaning up: Error migrating containers to aufs: Error rewriting container config for f6767d30d030f98813480bc41c26683d5ba91918331577767ca451a697f4920e: open /var/li>
Jun 23 14:40:47 a5fa2e0 systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE

Steps to reproduce the issue:

1. Perform a hostOS update from 2.72.0+rev1 to 2.95.8 on a balenaFin. Note that the error might need the exact application container images that this has been reproduced with.
2. On boot, check that the engine has failed to migrate.

Describe the results you received:

The engine fails to starts and keeps trying to migrate in a loop.

Describe the results you expected:
One of:

• The engine correctly migrates the application containers
• The engine fails to migrate the containers, but it removes the old ones and re-downloads new ones.

Additional information you deem important (e.g. issue happens only occasionally):

The issue happens with specific applications.

Output of balena-engine version:

Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.2
 Git commit:        73c78258302d94f9652da995af6f65a621fac918
 Built:             Wed Mar  2 18:29:36 2022
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.2
  Git commit:       73c78258302d94f9652da995af6f65a621fac918
  Built:            Wed Mar  2 18:29:36 2022
  OS/Arch:          linux/arm
  Experimental:     true
 containerd:
  Version:          1.4.0+unknown
  GitCommit:        
 runc:
  Version:          spec: 1.0.2-dev
  GitCommit:        
 balena-engine-init:
  Version:          0.13.0
  GitCommit:        949e6fa-dirty

Output of balena-engine info:

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 14
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host null
  Log: journald json-file local
 Swarm: 
  NodeID: 
  Is Manager: false
  Node Address: 
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: balena-engine-init
 containerd version: 
 runc version: 
 init version: 949e6fa-dirty (expected: de40ad007797e)
 Kernel Version: 5.10.83-v7
 Operating System: balenaOS 2.95.8
 OSType: linux
 Architecture: armv7l
 CPUs: 4
 Total Memory: 970.7MiB
 Name: a5fa2e0
 ID: DVU5:EUK4:ZANG:C4IA:DHRR:ZZI5:WW4Y:RK6C:CJRJ:Q3RW:JNLX:S6LQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional environment details (device type, OS, etc.):

BalenaFin running balenaOS 2.95.8

[jellyfish-bot]

[alexgg] This has attached https://jel.ly.fish/9aaf5f96-265c-4c52-9e59-074e973b53f8