Cannot boot debian image built with elemental-toolkit

[matteyeux]

elemental-toolkit version:
v2.2.1 (also tested with v2.2.2)

CPU architecture, OS, and Version:

Linux localhost 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux

Describe the bug
The VM fails to boot because /sysroot cannot be mounted, here is the log when the VM is booting :

[  OK  ] Finished dracut-pre-mount.…rvice - dracut pre-mount hook.
         Mounting sysroot.mount - /sysroot...
[    8.800735] overlayfs: failed to resolve '/run/overlayfs': -2
[FAILED] Failed to mount sysroot.mount - /sysroot.
See 'systemctl status sysroot.mount' for details.
[DEPEND] Dependency failed for elem…nitramfs setup before switch root.
[DEPEND] Dependency failed for elem…emental system early rootfs setup.
[  OK  ] Reached target initrd-root…get - Initrd Root File System.
         Starting initrd-parse-etc.…nts Configured in the Real Root...
[  OK  ] Finished initrd-parse-etc.…oints Configured in the Real Root.
[  OK  ] Reached target initrd-fs.target - Initrd File Systems.
[  OK  ] Reached target initrd.target - Initrd Default Target.
         Starting dracut-mount.service - dracut mount hook...
[    8.635974] dracut-mount[853]: Warning: Can't mount root filesystem
         Starting dracut-emergency.…ce - Dracut Emergency Shell...

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

To Reproduce

Use the following Dockerfile that uses debian:stable-slim : docker build -t matteyeux/debtest:test . (image is also available on the docker hub.

FROM ghcr.io/rancher/elemental-toolkit/elemental-cli:v2.2.1 AS toolkit

# OS base image of our choice
FROM debian:stable-slim AS os

ENV DEBIAN_FRONTEND=noninteractive \
    PYTHONUNBUFFERED=1 \
    TZ=Etc/UTC

RUN apt update && apt upgrade -y && apt install -y --no-install-recommends \
    linux-image-amd64 dmsetup dracut-core dracut-network dracut-live dracut-squash \
    grub2-common   grub-pc-bin   grub-efi-amd64 shim-signed haveged systemd systemd-sysv \
    systemd-timesyncd systemd-resolved openssh-server openssh-client tzdata parted e2fsprogs \
    dosfstools mtools xorriso findutils gdisk rsync squashfs-tools lvm2 vim less sudo ca-certificates \
    curl iproute2 dbus-daemon patch netplan.io locales kbd podman btrfs-progs btrfsmaintenance xz-utils \
    python3 python3-setuptools python3-pip vim file overlayroot

# Hack to prevent systemd-firstboot failures while setting keymap, this is known
# Debian issue (T_T) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=790955
ARG KBD=2.6.4
RUN curl -L https://mirrors.edge.kernel.org/pub/linux/utils/kbd/kbd-${KBD}.tar.xz --output kbd-${KBD}.tar.xz && \
    tar xaf kbd-${KBD}.tar.xz && mkdir -p /usr/share/keymaps && cp -Rp kbd-${KBD}/data/keymaps/* /usr/share/keymaps/

# Symlink grub2-editenv
RUN ln -sf /usr/bin/grub-editenv /usr/bin/grub2-editenv

# Just add the elemental cli
COPY --from=toolkit /usr/bin/elemental /usr/bin/elemental

# Enable essential services
RUN systemctl enable systemd-networkd.service

# Enable /tmp to be on tmpfs
RUN cp /usr/share/systemd/tmp.mount /etc/systemd/system

# Generate en_US.UTF-8 locale, this the locale set at boot by
# the default cloud-init
RUN locale-gen --lang en_US.UTF-8

# Generate initrd with required elemental services
RUN elemental --debug init -f

# Update os-release file with some metadata
RUN echo TIMESTAMP="`date +'%Y%m%d%H%M%S'`" >> /etc/os-release && \
    echo GRUB_ENTRY_NAME=\"Elemental\"      >> /etc/os-release

# Adding specific network configuration based on netplan
# ADD 05_network.yaml /system/oem/05_network.yaml
# Arrange bootloader binaries into /usr/lib/elemental/bootloader
# this way elemental installer can easily fetch them
RUN mkdir -p /usr/lib/elemental/bootloader && \
    cp /usr/lib/grub/x86_64-efi/monolithic/grubx64.efi /usr/lib/elemental/bootloader/grubx64.efi && \
    cp /usr/lib/shim/shimx64.efi.signed /usr/lib/elemental/bootloader/shimx64.efi && \
    cp /usr/lib/shim/mmx64.efi /usr/lib/elemental/bootloader/mmx64.efi

# Good for validation after the build
CMD ["/bin/bash"]

Push the image.

Then build the ISO : docker run --rm -v "$PWD:/output" ghcr.io/rancher/elemental-toolkit/elemental-cli:v2.2.1 --debug build-iso --bootloader-in-rootfs -n myiso matteyeux/debtest:test -o /output.

Boot it in VMware workstation, it will fail when booting.

Expected behavior
I expected it to boot

I tried to add a few packages etc... But I don't really know what I am doing. Any tip would be helpful. I tested with debian:stable (without slim), same result

[kkaempf]

Did you compare your Dockerfile with the sample Ubuntu one ?

[matteyeux]

Yes @kkaempf, the Dockerfile is based on the ubuntu example from this repo.

[frelon]

hi @matteyeux!

I get the same error as you when building and booting the ISO.

Raw disk works if you also install snapper and use the btrfs snapshotter (same as green example).

I think the problem is with the version of dracut, adding rd.live.overlay.overlayfs=1 to the kernel commandline enables to boot but I'm stuck at the login prompt.

I will continue to investigate and write here if I find anything!

[matteyeux]

@frelon thank you, I was able to get it boot. Tho based on the logs I still have the sysroot failure, but it does not seem block the boot process.

The root/elemental password does not work I will try using a custom cloud-init config to get a shell