Clarification on kata-containers support

[Fricounet]

Hello folks, I'm trying to wrap my head around how can nydus work wtih kata-containers.
I'm a bit confused because I can find 2 different references that indicate that nydus supports kata but in different ways:

• this doc in the kata project indicates that I need to configure enable_nydus_overlayfs in the nydus-snapshotter + use virtio-fs-nydus in kata
• this PR added a enable_kata_volume and a proxy mode to the snapshotter which seems to respectively pass the mounted image using EROFS or to let the guest VM mount the image

On the other hand, I tried pulling a nydus image in a kata container but without any of the above configured (so just a plain regular nydus config) and it just worked somehow. At least I couldn't notice a big issue and the image was pulled at regular nydus speed. So I'm really confused on whether all the above configs are needed at all?

I would really appreciate your thoughts on this and if you could indicate what's the difference and benefits/drawbacks for each approach.

For a bit of context on what I'm running:

• containerd v2.1.0 with bits to use the transfer service

version = 3

[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    # Required by the kubelet's imageServiceEndpoint
    address = "/run/containerd/image-service.sock"

  [proxy_plugins.nydus.exports]
    enable_remote_snapshot_annotations = "true"

[plugins]
  # Use nydus snapshotter through CRI
  [plugins."io.containerd.cri.v1.images"]
    snapshotter = "nydus"
    disable_snapshot_annotations = false
    discard_unpacked_layers = false

	[plugins."io.containerd.cri.v1.images".pinned_images]
  	# need sandbox_image entry here otherwise it's overwritten when importing the file
  	# sandbox_image is the image used by sandbox container.
  	sandbox = "pause"

  # Use nydus snapshotter through containerd transfer service
  [[plugins."io.containerd.transfer.v1.local".unpack_config]]
    snapshotter = "nydus"
    # need platform entry here otherwise there is no default used in containerd
    platform = "linux/amd64"

  # This is the default entry, which is removed if we specify another one, so we need to add it back explicitely
  [[plugins."io.containerd.transfer.v1.local".unpack_config]]
    snapshotter = "overlayfs"
    platform = "linux/amd64"
    differ = ""

• nydusd v2.3.1 with default config

{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "timeout": 15,
        "connect_timeout": 15,
        "retry_limit": 5
      }
    },
    "cache": {
      "type": "blobcache",
      "config": {
        "work_dir": "cache"
      }
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true,
  "fs_prefetch": {
    "enable": true,
    "threads_count": 4
  }
}

• nydus-snapshotter v0.15.0 with basic config + enable_referrer_detect

version = 1

[system]
# Snapshotter's debug and trace HTTP server interface
enable = true

[remote.auth]
# Fetch the private registry auth as CRI image service proxy
enable_cri_keychain = true
image_service_address = "/run/containerd/containerd.sock"

[experimental]
# Whether to enable referrers support
# The option enables trying to fetch the Nydus image associated with the OCI image and run it.
# Also see https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers
enable_referrer_detect = true

[imeoer]

Hi @Fricounet, as far as I know:

• enable_nydus_overlayfs = true: enable kata guest to access the rootfs of nydus image shared via the host's rootfs using nydusd daemon's virtiofs driver.
• fs_driver = "proxy" and enable_kata_volume = true: enable kata guest pull support, this allows containerd to skip downloading image layers on the host and instead pass the image reference label information to kata, let kata pull image in the guest.

cc @jiangliu

[Fricounet]

Hi @imeoer thanks!
This aligns with my understanding of those options.
What I wondering still, is if it is expected for nydus image pulls to work in in kata-containers even without any of those configured?
And if so, what is the actual benefit of using enable_nydus_overlayfs compared to not using anything at all?

[lukasmrtvy]

@fidencio friedly ping :) Is enable_nydus_overlayfs really required as stated here https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/how-to-use-virtio-fs-nydus-with-kata.md

Thanks