runc no_pivot test fails on ppc64le

[ricardobranco777]

Description

runc no_pivot test fails on ppc64le.

Tracking in SUSE as https://bugzilla.suse.com/show_bug.cgi?id=1247568

openQA job: https://openqa.opensuse.org/tests/5220249

Steps to reproduce the issue

### RUN AS root
curl -sL --retry 9 --retry-delay 100 --retry-max-time 900 https://github.com/bats-core/bats-core/archive/refs/tags/v1.11.1.tar.gz | tar -zxf -
bash bats-core-1.11.1/install.sh /usr/local
rm -rf bats-core-1.11.1
mkdir -pm 0750 /etc/sudoers.d/
echo 'Defaults secure_path="/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin"' > /etc/sudoers.d/usrlocal
zypper --gpg-auto-import-keys -n install glibc-devel-static go1.24 jq libseccomp-devel make runc criu
mkdir -p /etc/systemd/system/user@.service.d/
echo -e "[Service]\nDelegate=cpu cpuset io memory pids" > /etc/systemd/system/user@.service.d/60-delegate.conf
reboot
### RUN AS user
cd /var/tmp/
git clone https://github.com/opencontainers/runc.git
cd /var/tmp/runc
git checkout v1.3.0
make memfd-bind fs-idmap pidfd-kill recvtty remap-rootfs sd-helper seccompagent || true
### RUN AS root
cd /var/tmp/runc
mkdir -p /var/tmp/test.qFQ71a
env BATS_TMPDIR=/var/tmp/test.qFQ71a PATH=/usr/local/bin:$PATH:/usr/sbin:/sbin RUNC=/usr/bin/runc RUNC_USE_SYSTEMD=1 bats --tap -T tests/integration/no_pivot.bats
sudo rm -rf /var/tmp/test.qFQ71a || true

Describe the results you received and expected

not ok 174 runc run --no-pivot must not expose bare /proc in 961ms
# (in test file tests/integration/no_pivot.bats, line 21)
#   `[ "$status" -eq 1 ]' failed
# runc spec (status=0):
#
# runc run --no-pivot test_no_pivot (status=139):
# + mount -t proc none /proc
# mount: permission denied (are you root?)
# --- teardown ---

What version of runc are you using?

runc version 1.3.0
commit: v1.3.0-0-g4ca628d1d4c9
spec: 1.2.1
go: go1.24.2
libseccomp: 2.6.0

Host OS information

openSUSE Tumbleweed 20250803

Host kernel information

Linux opensuse 6.15.8-1-default #1 SMP PREEMPT_DYNAMIC Thu Jul 24 07:19:58 UTC 2025 (e03d052) x86_64 x86_64 x86_64 GNU/Linux

[cyphar]

Are you replacing BUSYBOX_IMAGE with some other rootfs? Our scripts (tests/integration/get-images.sh) download the archives from the internet but I suspect that isn't allowed in openQA scripts.

We do expect an error here, and the error message is correct -- but the error code is 139 not 1. The most obvious cause I can think of would be that the mount (or unshare) binaries are not the ones we use in our tests, and so have different error codes.

Or maybe there's something up with the ppc64le binaries we use? We don't have CI for some of the more "esoteric" architectures like ppc64le and s390x. I think that the only time I've managed to test these was using SUSE equipment internally...

[ricardobranco777]

Are you replacing BUSYBOX_IMAGE with some other rootfs? Our scripts (tests/integration/get-images.sh) download the archives from the internet but I suspect that isn't allowed in openQA scripts.

We don't call this script directly and I guess it's called by helpers.bash.

We do expect an error here, and the error message is correct -- but the error code is 139 not 1. The most obvious cause I can think of would be that the mount (or unshare) binaries are not the ones we use in our tests, and so have different error codes.

Will look into this.

[ricardobranco777]

We do expect an error here, and the error message is correct -- but the error code is 139 not 1. The most obvious cause I can think of would be that the mount (or unshare) binaries are not the ones we use in our tests, and so have different error codes.

Or a SIGSEGV. I see a call trace here:

https://openqa.opensuse.org/tests/5220249/file/serial0.txt

[kolyshkin]

Or a SIGSEGV. I see a call trace here:

https://openqa.opensuse.org/tests/5220249/file/serial0.txt

The call trace might be from the OOM killer, there are some tests that cause an OOM, so unless you're running this test separately, this might be unrelated.

You can run specific bats tests by supplying either file name(s) or -f <test-name-regex> to bats.

[kolyshkin]

In case it's SIGSEGV (looks like it), a stack trace will be helpful.

[ricardobranco777]

I ran this job running only this test: https://openqa.opensuse.org/tests/5224428

Please tell me if you need any more information before the garbage collector removes some logs. I can restart the test anyway.

Output of journalctl here: https://openqa.opensuse.org/tests/5224428/file/runc-journalctl-b.txt

Aug 06 04:14:54 susetest systemd[1]: Started libcontainer container test_no_pivot.
Aug 06 04:14:54 susetest kernel: unshare[3528]: segfault (11) at 4699b2e8eead9efc nip 4699b2e8eead9efc lr 126be4fe8 code 3
Aug 06 04:14:54 susetest kernel: unshare[3528]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX 
Aug 06 04:14:54 susetest kernel: unshare[3528]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX 
Aug 06 04:14:54 susetest systemd-coredump[3536]: Process 3528 (unshare) of user 0 terminated abnormally with signal 11/SEGV, processing...
Aug 06 04:14:54 susetest systemd[1]: Created slice Slice /system/systemd-coredump.
Aug 06 04:14:54 susetest systemd[1]: Started Process Core Dump (PID 3536/UID 0).
Aug 06 04:14:54 susetest systemd-coredump[3537]: Process 3528 (unshare) of user 0 dumped core.
                                                 
                                                 Stack trace of thread 1:
                                                 #0  0x4699b2e8eead9efc n/a (n/a + 0x0)
                                                 #1  0x0000000126be4fe8 n/a (/bin/unshare + 0x14fe8)
                                                 #2  0x0000000126be6520 n/a (/bin/unshare + 0x16520)
                                                 #3  0x0000000126be65a0 n/a (/bin/unshare + 0x165a0)
                                                 #4  0x0000000126c8819c n/a (/bin/unshare + 0xb819c)
                                                 #5  0x0000000126be31e0 n/a (/bin/unshare + 0x131e0)
                                                 #6  0x0000000126be3684 n/a (/bin/unshare + 0x13684)
                                                 #7  0x0000000126be3748 n/a (/bin/unshare + 0x13748)
                                                 #8  0x00007fff8d094bcc n/a (/lib/libc.so.6 + 0x24bcc)
                                                 #9  0x00007fff8d094db0 n/a (/lib/libc.so.6 + 0x24db0)
                                                 ELF object binary architecture: PowerPC64

[cyphar]

If you can grab the coredump and provide it to me (either here or slack), that could be useful. Though I suspect that our busybox images don't have any debug symbols...

[ricardobranco777]

If you can grab the coredump and provide it to me (either here or slack), that could be useful. Though I suspect that our busybox images don't have any debug symbols...

You can download it from here:
https://openqa.opensuse.org/tests/5224801/file/runc-unshare.3494.core.gz

[kolyshkin]

So, we are using a somewhat old version of busybox:glibc in our tests (1.34.1). Might make sense to bump to 1.35.1, although I just realized I have no idea what to put into https://github.com/opencontainers/runc/blob/main/tests/integration/bootstrap-get-images.sh#L13 to make it happen. @tianon can you help 🥺🙏🏻

[ricardobranco777]

There's also kernel call traces in the original job:

https://openqa.opensuse.org/tests/5220249/file/serial0.txt

[ricardobranco777]

There's also kernel call traces in the original job:

https://openqa.opensuse.org/tests/5220249/file/serial0.txt

Those can be ignored as they were triggered by the OOM killer.

[vishalc-ibm]

I don’t think this issue has anything to do with the OOM killer either.

Following the steps shared by @ricardobranco777, I was able to reproduce the problem on a local setup. Every time I ran the no_pivot testcase, I consistently saw the following kernel logs:

[  357.237289][ T6093] unshare[6093]: segfault (11) at 4699b2e8eead9efc nip 4699b2e8eead9efc lr 1161a4fe8 code 3
[  357.237331][ T6093] unshare[6093]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX
[  357.237338][ T6093] unshare[6093]: code: XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX

Narrowing down the problem

I tried changing the testcase parameters:

• Removing the --no-pivot option from runc run → the segfault still occurred.
• Tweaking the arguments passed to unshare → when I removed -f from -mrpf, the segfault disappeared.

This pointed to an issue with the unshare binary itself.

Checking the unshare version

I modified the testcase to run unshare -V instead:

-       update_config '   .process.args |= ["unshare", "-mrpf", "sh", "-euxc", "mount -t proc none /proc && echo h > /proc/sysrq-trigger"]
+       update_config '   .process.args |= ["unshare", "-V"]
                        | .process.capabilities.bounding += ["CAP_SETFCAP"]
                        | .process.capabilities.permitted += ["CAP_SETFCAP"]'

This showed that the unshare used in the testcase was v1.34.1, while the system’s native unshare was v2.32.1.

Running the command directly with the system’s unshare worked fine:

unshare -mrpf sh -euxc 'mount -t proc none /proc && echo h > /proc/sysrq-trigger'

So the issue seemed isolated to BusyBox’s version of unshare.

Debugging further

I collected a coredump from the crashing BusyBox unshare, but without debug symbols the backtrace was useless.

Next, I rebuilt BusyBox with debug symbols and re-ran the same command. This time, the segfault reproduced even without involving runc.

The backtrace was:

(gdb) bt
#0  0x00000000100080e4 in 0000004f.plt_call.waitpid@@GLIBC_2.17 ()
#1  0x000000001000dbfc in safe_waitpid (pid=<optimized out>, wstat=0x7fffdf2ef8c0, options=<optimized out>) at libbb/xfuncs.c:398
#2  0x000000001000f218 in wait_for_exitstatus (pid=<optimized out>) at libbb/xfuncs_printf.c:721
#3  0x000000001000f298 in xvfork_parent_waits_and_exits () at libbb/xfuncs_printf.c:707
#4  0x00000000100b27d0 in unshare_main (argc=<optimized out>, argv=<optimized out>) at util-linux/unshare.c:376
#5  0x000000001000bde4 in run_applet_no_and_exit (applet_no=<optimized out>, name=name@entry=0x7fffdf2ff525 "unshare", argv=argv@entry=0x7fffdf2eff58)
    at libbb/appletlib.c:971
#6  0x000000001000c260 in run_applet_and_exit (name=0x7fffdf2ff525 "unshare", argv=argv@entry=0x7fffdf2eff58) at libbb/appletlib.c:990
#7  0x000000001000c328 in main (argc=<optimized out>, argv=0x7fffdf2eff58) at libbb/appletlib.c:1130

Conclusion

The segfault is not related to runc or the kernel. The problem lies in BusyBox’s implementation of unshare when used with -mrpf. The system’s unshare binary works correctly, so this looks like a BusyBox bug.