Issues running a nydus image, pull works fine but run fails: "ctr: failed to find image ref of snapshot <X>..."

[KristoferHansson]

I have tried getting nydus snapshotter to work for a while now but yet to be successful, I have posted question in nerdctl github and nydus slack but still not been able to get it to work:

My goal is to verify chunk level deduplication efficiency but I'm still stuck on even getting a first nydus image running

I'm building 5 images, pushing them to a local hosted registry (tested with registry:v2 and zot, running as a container), converting them using nydusify (pull, convert, push). Then in a separate distro I successfully pull the nydus images (using ctr and nerdctl), then when I try to run them it fails.

sudo ./ctr --address /run/containerd/containerd.sock run --rm --snapshotter nydus --platform linux/amd64 --net-host localhost:5000/testimg5-nydusss:nydus test1 sh -c "find / > /dev/null"
WARN[0000] DEPRECATION: The `mirrors` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.1.Use `config_path` instead.
WARN[0000] DEPRECATION: The `configs` property of `[plugins."io.containerd.grpc.v1.cri".registry]` is deprecated since containerd v1.5 and will be removed in containerd v2.2.Use `config_path` instead.
ctr: failed to find image ref of snapshot 4, labels map[containerd.io/snapshot.ref:sha256:8895493b76342d2901ffa219fa0703f67fd5c3b0e97f686eced3e85809b93362 containerd.io/snapshot/nydus-bootstrap:true containerd.io/snapshot/nydus-fs-version:6]

The snapshotter output:

INFO[2025-09-06T17:59:52.903783581+02:00] [Prepare] snapshot with key default/8/test1 parent default/5/sha256:8895493b76342d2901ffa219fa0703f67fd5c3b0e97f686eced3e85809b93362
DEBU[2025-09-06T17:59:52.907116271+02:00] [Prepare] snapshot with labels map[]          key=default/8/test1 parent="default/5/sha256:8895493b76342d2901ffa219fa0703f67fd5c3b0e97f686eced3e85809b93362"
DEBU[2025-09-06T17:59:52.907191977+02:00] isProxyDriver = false, isProxyLabel = false, isProxyImage = false
INFO[2025-09-06T17:59:52.907218583+02:00] Prepare active Nydus snapshot default/8/test1  key=default/8/test1 parent="default/5/sha256:8895493b76342d2901ffa219fa0703f67fd5c3b0e97f686eced3e85809b93362"
DEBU[2025-09-06T17:59:52.907240878+02:00] Prepare remote snapshot 4                     key=default/8/test1 parent="default/5/sha256:8895493b76342d2901ffa219fa0703f67fd5c3b0e97f686eced3e85809b93362"

note: this generates no additional zot image registry output

looking at the manifest and checking for the existence of the blobs locally shows than only the last blob in the manifest of each image exists locally:

❯ for d in (no_proxy=localhost curl -s http://localhost:5000/v2/testimg5-nydusss/manifests/nydus | jq -r '.layers[].digest' | sed 's/^sha256://')                      sudo ls -lh /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/$d
  end
ls: cannot access '/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/98b707aea280e075d94c89676551b8fbd5a6097cfab3f566ebb5b8130c21830b': No such file or directory
ls: cannot access '/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/cd944a11efae876b5d937e09c8d4542851ddff8726fae4db2084dcc3fa605395': No such file or directory
ls: cannot access '/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/809abd7a52aca74ff2ac4b297a9396d2a85feffaea2e30f38a74695da858a6e8': No such file or directory
-r--r--r-- 1 root root 216K Sep  6 17:43 /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256/a991aa3a0d6cbfc6a9f8b19a947de52188aa52104cef99fd38999960cb54fe9b

but they exist in the registry:

❯ for d in (no_proxy=localhost curl -s http://localhost:5000/v2/testimg5-nydusss/manifests/nydus | jq -r '.layers[].digest')
          no_proxy=localhost curl -I http://localhost:5000/v2/testimg5-nydusss/blobs/$d
  end
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 10494225
Docker-Content-Digest: sha256:98b707aea280e075d94c89676551b8fbd5a6097cfab3f566ebb5b8130c21830b
Date: Sat, 06 Sep 2025 16:41:47 GMT

HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 10494215
Docker-Content-Digest: sha256:cd944a11efae876b5d937e09c8d4542851ddff8726fae4db2084dcc3fa605395
Date: Sat, 06 Sep 2025 16:41:47 GMT

HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 32165106
Docker-Content-Digest: sha256:809abd7a52aca74ff2ac4b297a9396d2a85feffaea2e30f38a74695da858a6e8
Date: Sat, 06 Sep 2025 16:41:47 GMT

HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 220623
Docker-Content-Digest: sha256:a991aa3a0d6cbfc6a9f8b19a947de52188aa52104cef99fd38999960cb54fe9b
Date: Sat, 06 Sep 2025 16:41:47 GMT

References:
containerd/nerdctl#4495
https://nydusimageservice.slack.com/archives/C01RUGPTP37/p1756467674990349

[imeoer]

Hi @KristoferHansson, the ctr does not support running nydus image, as it not to pass necessary image info labels to the remote snapshotter, try using nerdctl instead.

[BraveY]

Hi, Could you try the nydusify check --source [oci-image] --target [nydus-image] command to validate the nydus images?

looking at the manifest and checking for the existence of the blobs locally shows than only the last blob in the manifest of each image exists locally:

What is the media type of last blobs? It should be the bootstrap layer, You can refer to the format of the bootstrap layer in the example。

    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:aec98c9e3dce739877b8f5fe1cddd339de1db2b36c20995d76f6265056dbdb08",
      "size": 273320,
      "annotations": {
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-reference-blob-ids": "[\"09845cce1d983b158d4865fc37c23bbfb892d4775c786e8114d3cf868975c059\"]"
      }
    }

Nydus only pulls the bootstrap layer first; other data layers are downloaded through prefetching or on-demand(lazy load) pulling after the pod starts, So this is reasonable。

[KristoferHansson]

Hi, thanks for quick responses, below are the results of the proposed checks,

Edit: Pasted the results of testimage1 below, when in previous post having used testimage5, I re-verified with both now just to be sure there was no problem with that, results are "same".

@imeoer: ok, started trying with ctr to check all parts of the image prep after having failed with nerdctl, but probably just got carried along, given that it did not complain about the --snapshotter nydus argument, but getting basically the exact same errors with nerdctl.

@BraveY: "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", so not the same type as in your example

❯ no_proxy=localhost curl -s http://localhost:5000/v2/testimg1-nydusss/manifests/nydus | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {                                                                                                                                                    "mediaType": "application/vnd.docker.container.image.v1+json",
    "digest": "sha256:029b6df1b195aad4a64d9dae048a65979460fa46de0ac1dcd12821c267e187fb",
    "size": 1825
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:7c857b86ffc9da4b9f11cfddf4d85d88bf3ebb3f16e49cecf6ff8779f3cc5611",
      "size": 10494226,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:cd944a11efae876b5d937e09c8d4542851ddff8726fae4db2084dcc3fa605395",
      "size": 10494215,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:809abd7a52aca74ff2ac4b297a9396d2a85feffaea2e30f38a74695da858a6e8",
      "size": 32165106,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "digest": "sha256:f3d54fb4a86cda4fb6eb7864683830156d25e0be74ca8feee2175323c6ad5122",
      "size": 220622,
      "annotations": {
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-fs-version": "6"
      }
    }
  ],
  "annotations": {
    "containerd.io/snapshot/nydus-builder-version": "v2.3.4",
    "containerd.io/snapshot/nydus-fs-version": "6",
    "containerd.io/snapshot/nydus-source-digest": "sha256:3b0e5063b589d1a00cebe77822128855d47fa7bc6f62e7a82a2d08d625ffd11a",
    "containerd.io/snapshot/nydus-source-reference": "localhost:5000/testimg1:raw"
  }
}

nydusify check seems to think it's ok.

INFO[2025-09-08T09:02:51+02:00] parsing image                                 image="localhost:5000/testimg1-nydusss:nydus"
INFO[2025-09-08T09:02:51+02:00] fetch failed                                  error="failed to do request: Head \"https://localhost:5000/v2/testimg1-nydusss/manifests/nydus\": http: server gave HTTP response to HTTPS client" host="localhost:5000"
INFO[2025-09-08T09:02:51+02:00] parsing image                                 image="localhost:5000/testimg1-nydusss:nydus"
INFO[2025-09-08T09:02:51+02:00] dumping manifest                              image="localhost:5000/testimg1:raw" type=oci
INFO[2025-09-08T09:02:51+02:00] dumping manifest                              image="localhost:5000/testimg1-nydusss:nydus" type=nydus
INFO[2025-09-08T09:02:51+02:00] pulling bootstrap                             image="localhost:5000/testimg1-nydusss:nydus" type=nydus
INFO[2025-09-08T09:02:51+02:00] checking manifest                             image="localhost:5000/testimg1:raw" type=oci
INFO[2025-09-08T09:02:51+02:00] checking manifest                             image="localhost:5000/testimg1-nydusss:nydus" type=nydus
INFO[2025-09-08T09:02:51+02:00] checking bootstrap                            image="localhost:5000/testimg1-nydusss:nydus" type=nydus
RAFS filesystem metadata is valid, referenced data blobs:
         0: 7c857b86ffc9da4b9f11cfddf4d85d88bf3ebb3f16e49cecf6ff8779f3cc5611, compressed size 10491764, uncompressed size 10485760, chunk size: 1048576, chunk count: 10, features: aligned cap_toc chunk-digest tar-header toc
         1: cd944a11efae876b5d937e09c8d4542851ddff8726fae4db2084dcc3fa605395, compressed size 10491764, uncompressed size 10485760, chunk size: 1048576, chunk count: 10, features: aligned cap_toc chunk-digest tar-header toc
         2: 809abd7a52aca74ff2ac4b297a9396d2a85feffaea2e30f38a74695da858a6e8, compressed size 31953903, uncompressed size 83820544, chunk size: 1048576, chunk count: 2516, features: aligned cap_toc chunk-digest tar-header toc
INFO[2025-09-08T09:02:51+02:00] mounting image                                image="localhost:5000/testimg1:raw" type=oci
INFO[2025-09-08T09:02:52+02:00] mounting image                                image="localhost:5000/testimg1-nydusss:nydus" type=nydus
INFO[2025-09-08T09:02:52+02:00] comparing filesystem
INFO[2025-09-08T09:02:53+02:00] verified image

[BraveY]

"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", so not the same type as in your example

Ok, your image is a rafs v6 version. The example is rafs v5.

nydusify check command passed mean your image is a valid nydus image.
Please give detail steps to reproduce your issue.