[Bug] nydus container can't read same file #1774
Description
Problem Description
the following message is the log of nydusd:
Aug 31 22:19:04 node.name containerd-nydus-grpc[2242137]: out_opts: ASYNC_READ | BIG_WRITES | HAS_IOCTL_DIR | AUTO_INVAL_DATA | DO_READDIRPLUS | READDIRPLUS_AUTO | ASYNC_DIO | WRITEBACK_CACHE | ZERO_MESSAGE_OPEN | PARALLEL_DIROPS
Aug 31 22:19:04 node.name containerd-nydus-grpc[2242137]: [2025-08-31 22:19:04.095793 +08:00] INFO Fuse daemon started!
Aug 31 22:19:19 node.name containerd-nydus-grpc[2242137]: [2025-08-31 22:19:19.401968 +08:00] ERROR [/src/error.rs:22] Error:
Aug 31 22:19:19 node.name containerd-nydus-grpc[2242137]: "failed to prefetch data from storage backend for chunk 37/3"
Aug 31 22:19:19 node.name containerd-nydus-grpc[2242137]: at storage/src/cache/cachedfile.rs:791
---
Mar 03 19:32:04 containerd-nydus-grpc[1569547]: [2025-03-03 19:32:04.380846 +08:00] ERROR [/src/error.rs:22] Error:
Mar 03 19:32:04 containerd-nydus-grpc[1569547]: "failed to read metadata for blob fe5b9741c280595c052ca097b213d08593210f112943cd70f11d7cb180431203 from backend, Request(Common(reqwest::Error { kind: Request, url: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("registry.corp.kuaishou.com")), port: None, path: "/v2/ksp/ad-kuaishou-e-runner/blobs/sha256:fe5b9741c280595c052ca097b213d08593210f112943cd70f11d7cb180431203", query: None, fragment: None }, source: TimedOut }))"
Mar 03 19:32:04 containerd-nydus-grpc[1569547]: at storage/src/meta/mod.rs:676
Mar 03 19:32:04 containerd-nydus-grpc[1569547]: note: enable RUST_BACKTRACE=1 env to display a backtrace
Mar 03 19:32:04 containerd-nydus-grpc[1569547]: [2025-03-03 19:32:04.380911 +08:00] INFO temporarily failed to get blob.meta, Input/output error (os error 5)
Expected Behavior
failed to read file, then failed to run the container.
Actual Behavior
read all the image file.
How to reproduce
This is a heisenbug - it disappears when we try to debug it.
Environment Details
- Nydus-snapshotter version: stable/0.9
- Nydus version: stable/2.2.3
- Container runtime: 1.7
- Operating System: centos7
- Kernel version: 4.6.2
Additional Information
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!