containerd mount callback failed

[mfrankm]

Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount4004440706: read /var/lib/containerd/tmpmounts/containerd-mount4004440706/etc/passwd: invalid argument
The error information is as above. When k8s starts the nydus image pod, the above error will be reported. The versions are nydus-snapshotter-v0.15.0 and nydus-static-v2.3.0. What is the reason and how to solve it?

[criscola]

Same issue here.

[imeoer]

Hi @criscola please retry with the release v2.3.4: dragonflyoss/nydus#1720 (comment)

[xoJIog]

Hello!

i have upgraded to 2.3.4 and problem persists

Error: failed to create containerd container: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1441598952: read /var/lib/containerd/tmpmounts/containerd-mount1441598952/etc/passwd: invalid argument

when i was using nydus-snapshotter with standard docker images - there was no problem, but when i converted images using nydusify - the problem started

i tried converting images to nydus format as well as using --oci flag

nydus-snapshotter config:

version = 1
root = "/var/lib/containerd/io.containerd.snapshotter.v1.nydus"
address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
daemon_mode = "dedicated"
cleanup_on_close = false

[system]
enable = false
address = "/run/containerd-nydus/system.sock"

[system.debug]
daemon_cpu_profile_duration_secs = 5
pprof_address = ""

[daemon]
nydusd_config = "/etc/nydus/nydusd-config.fusedev.json"
nydusd_path = "/usr/local/bin/nydusd"
nydusimage_path = "/usr/local/bin/nydus-image"
fs_driver = "fusedev"
recover_policy = "restart"
threads_number = 10
log_rotation_size = 100

[cgroup]
enable = true
memory_limit = ""

[log]
log_to_stdout = false
level = "info"
log_rotation_compress = true
log_rotation_local_time = true
log_rotation_max_age = 7
log_rotation_max_backups = 5
log_rotation_max_size = 100

[metrics]
address = ""

[remote]
convert_vpc_registry = false

[remote.mirrors_config]

[remote.auth]
enable_kubeconfig_keychain = false
kubeconfig_path = ""
enable_cri_keychain = false

[snapshot]
enable_nydus_overlayfs = false
nydus_overlayfs_path = "nydus-overlayfs"
enable_kata_volume = false
sync_remove = false

[cache_manager]
disable = false
gc_period = "24h"
cache_dir = ""

[image]
public_key_file = ""
validate_signature = false

[experimental]
enable_stargz = false
enable_referrer_detect = false
enable_backend_source = false
[experimental.tarfs]
enable_tarfs = false
mount_tarfs_on_host = false
tarfs_hint = false
max_concurrent_proc = 0
export_mode = ""

nydusd config:

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "scheme": "",
        "skip_verify": true,
        "timeout": 5,
        "connect_timeout": 5,
        "retry_limit": 4,
        "auth": ""
      }
    },
    "cache": {
      "type": "blobcache",
      "config": {
        "work_dir": "cache"
      }
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true,
  "fs_prefetch": {
    "enable": true,
    "threads_count": 4
  }
}

I tried enabling enable_nydus_overlayfs, and localfs for nydusd - on luck
nydusd version: v2.3.4
containerd-nydus-grpc version: v0.15.2
containerd version: 1.7.24

[xoJIog]

my problem was caused by nydusd not finding the blobs needed and was resolved by adding mirror in nydusd.json config where to search for blobs

[BraveY]

my problem was caused by nydusd not finding the blobs needed and was resolved by adding mirror in nydusd.json config where to search for blobs

Hi, in the next version, we’ve decided to remove the mirror configuration. Would you mind trying the proxy instead?

[xoJIog]

I am having trouble figuring it out, can you provide an example of nydusd config for use with Harbor that has multiple projects as proxy-cache ?

[BraveY]

I am having trouble figuring it out, can you provide an example of nydusd config for use with Harbor that has multiple projects as proxy-cache ?

The proxy backend temple example may be:

// for remote usage
{
  "device": {
      "backend": {
      "type": "http-proxy",
      "config": {
        "addr": "http://127.0.0.1:9977",
        "path": "/namespace/<repo>/blobs"
      }
    }
  }
}

[Zephyrcf]

I am having trouble figuring it out, can you provide an example of nydusd config for use with Harbor that has multiple projects as proxy-cache ?

The proxy backend temple example may be:

// for remote usage
{
  "device": {
      "backend": {
      "type": "http-proxy",
      "config": {
        "addr": "http://127.0.0.1:9977",
        "path": "/namespace/<repo>/blobs"
      }
    }
  }
}

It seems that nydus-snapshotter does not currently support docking with http-proxy.
@xoJIog I can't reproduce your problem. If harbor is authenticated locally, we should not need to configure nydusd config specifically (although harbor contains multiple projects as proxy cache)

[xoJIog]

Maybe i phrased the question wrong
I have a harbor service configured as proxy-cache for multiple remote repositories, I have a webhook setup on harbor to convert all images to nydus format. All worker nodes have containerd configured to go to harbor as a mirror for any original repository, containerd is also configured to use nydus snapshotter plugin.
The question is - which type of configuration would i require for nydusd? Really - no configuration?

[Zephyrcf]

Maybe i phrased the question wrong I have a harbor service configured as proxy-cache for multiple remote repositories, I have a webhook setup on harbor to convert all images to nydus format. All worker nodes have containerd configured to go to harbor as a mirror for any original repository, containerd is also configured to use nydus snapshotter plugin. The question is - which type of configuration would i require for nydusd? Really - no configuration?

Is there any error in the nydusify convert process triggered by this webhook? I think the key lies in the webhook stage.

[criscola]

my problem was caused by nydusd not finding the blobs needed and was resolved by adding mirror in nydusd.json config where to search for blobs

Hi, in the next version, we’ve decided to remove the mirror configuration. Would you mind trying the proxy instead?

It would be great if the docs is updated, in particular this page has still the old mirror config. And actually prompts the question, how does Dragonfly know about which registry should it forward to if there's not a headers config stanza anymore for proxy where X-Dragonfly-Registry was passed?

[imeoer]

@criscola Thanks for reminding about the doc updates, proxy mode uses HTTP tunnel to request dragonfly.