cpToPod failed after 30 attempts (related to permission issue)

[vvanouytsel]

I have been trying to implement #244 by using version 0.8.0 of runner-container-hooks.

However I have been struggling so far to get it to work.
This is the template I have been using.

template:
  spec:
    securityContext:
        fsGroup: 1000
    serviceAccountName: xxx-redacted
    containers:
    - name: runner
      image: my-company.registry.com/arc-linux-runner:2.328.0-3dacb977f4376955e068080cf4e8a42efe84e83d
      command: ["/home/runner/run.sh"]
      resources:
        limits:
          ephemeral-storage: 2Gi
          memory: 512Mi
        requests:
          cpu: 50m
          ephemeral-storage: 2Gi
          memory: 256Mi
      env:
        - name: ACTIONS_RUNNER_USE_KUBE_SCHEDULER
          value: "true"
        - name: ACTIONS_RUNNER_PREPARE_JOB_TIMEOUT_SECONDS
          value: "1800"
        - name: ACTIONS_RUNNER_CONTAINER_HOOKS
          value: /home/runner/k8s/index.js
        - name: ACTIONS_RUNNER_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
          value: "true"
        - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
          value: /home/runner/hook-extension.yaml
      volumeMounts:
        - name: hook-extension
          mountPath: /home/runner/hook-extension.yaml
          subPath: content
    nodeSelector:
      kubernetes.io/arch: amd64
    volumes:
      - name: hook-extension
        configMap:
          name: hook-extension-{{ gha_runner_scale_set__name }}

The Initialize Containers step always fails with the following message:

Run '/home/runner/k8s/index.js'
(node:66) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Error: Error: cpToPod failed after 30 attempts: {}
Error: Process completed with exit code 1.
Error: Executing the custom container implementation failed. Please contact your self hosted runner administrator

By enabling debug mode it becomes clear that the issue is related to permissions.

##[debug]cpToPod: Attempt 1 failed: Error: Error from cpFromPod - details: 
##[debug] tar: _PipelineMapping: Cannot mkdir: Permission denied
##[debug]tar: _temp: Cannot mkdir: Permission denied
##[debug]tar: _tool: Cannot mkdir: Permission denied
##[debug]tar: tm-pnt-dummy: Cannot utime: Operation not permitted
##[debug]tar: _PipelineMapping: Cannot mkdir: Permission denied
##[debug]tar: _PipelineMapping/TrendMiner: Cannot mkdir: No such file or directory
##[debug]tar: _temp: Cannot mkdir: Permission denied
##[debug]tar: _temp/_github_home: Cannot mkdir: No such file or directory
##[debug]tar: _temp: Cannot mkdir: Permission denied
##[debug]tar: _temp/_github_workflow: Cannot mkdir: No such file or directory
##[debug]tar: _temp: Cannot mkdir: Permission denied
##[debug]tar: _temp/_runner_hook_responses: Cannot mkdir: No such file or directory
##[debug]tar: tm-pnt-dummy/tm-pnt-dummy: Cannot utime: Operation not permitted
##[debug]tar: _PipelineMapping/TrendMiner: Cannot mkdir: No such file or directory
##[debug]tar: _PipelineMapping/TrendMiner/tm-pnt-dummy: Cannot mkdir: No such file or directory
##[debug]tar: _temp/_runner_hook_responses: Cannot mkdir: No such file or directory
##[debug]tar: _temp/_runner_hook_responses/aa549ea6-b227-441e-ab3f-7ca69acc9602.json: Cannot open: No such file or directory
##[debug]tar: _PipelineMapping/TrendMiner: Cannot mkdir: No such file or directory
##[debug]tar: _PipelineMapping/TrendMiner/tm-pnt-dummy/PipelineFolder.json: Cannot open: No such file or directory
##[debug]tar: .: Cannot utime: Operation not permitted
##[debug]tar: Exiting with failure status due to previous errors

The workflow that I am using is the following.

name: Troubleshoot autoscaling
on:
  workflow_dispatch:
jobs:
  job1:
    runs-on: default-staging
    container: company.registry.com/tm-gha-base:v1
    steps:
      - name: Use a big runner to test autoscaler
        run: sleep 600

Our tm-gha-base container is an ubuntu based container which runs as uid 1000 and thus not as root.
As you can see in the scaleset configuration, we are also defining a securityContext with fsGroup set to 1000.

Is there any support to use non-root containers, like we are using now?

[nikola-jokic]

Hey @vvanouytsel,

Runner's user is usually 1001. Can you please try that?

[vvanouytsel]

Hey @vvanouytsel,

Runner's user is usually 1001. Can you please try that?

Thanks for your response @nikola-jokic.

In our case the runner uses uid 1000 and guid 1000.

runner@default-staging-wjvgq-runner-dq5t9:~$ whoami
runner
runner@default-staging-wjvgq-runner-dq5t9:~$ id -u
1000
runner@default-staging-wjvgq-runner-dq5t9:~$ id -g
1000

Is there a way I can overwrite the uid for the initcontainer that performs the copy?
We are currently using the following hook extension:

spec:
  serviceAccountName: sa-gha-runner
  containers:
    - name: $job
      resources:
        requests:
          cpu: "500m"
          memory: 512Mi
        limits:
          memory: 512Mi
      imagePullPolicy: Always

[nikola-jokic]

Can you please try running in let's say ubuntu:latest container where the runner is root?

[vvanouytsel]

Can you please try running in let's say ubuntu:latest container where the runner is root?

When using ubuntu:latest as container it works.
When using any container that runs as root, it also works.

 # This works, because alpine:latest runs as root
  test-custom-runner-with-container:
    runs-on: default-staging
    container:
      image: my-company.registry.com/alpine:latest
    steps:
      - name: List
        run: |
          id -u
          id -g
          ls -l /
          ls -ld /__w

However, when we use our self-hosted runner in combination with a container that does not run as root, the issue occurs.
We've also started using 1001 for our runner (and reflected that via the securityContext as well) but the issue still persists.

We noticed that the /__w directory is always owned by root:root, which seems to be the main issue.

This is the output when we are using ubuntu-latest as runner.
You can see that the __w directory is properly owned by 1001.

Run id -u
##[debug]sh -e /__w/_temp/814bd0a7-90ca-4c89-a0ed-b9ea1077662a.sh
1001
1001
total 59216
drwxr-xr-x    6 runner runner     4096 Aug 13 16:46 __e
drwxrwxrwx+   9 runner root       4096 Sep 29 21:38 __t
drwxr-xr-x    6 runner runner     4096 Oct 13 09:07 __w
lrwxrwxrwx    1 root   root          7 Apr 22  2024 bin -> usr/bin
drwxr-xr-x    2 root   root       4096 Apr 22  2024 boot
drwxr-xr-x    5 root   root        340 Oct 13 09:07 dev
drwxr-xr-x    1 root   root       4096 Oct 13 09:07 etc
drwxr-xr-x    4 root   root       4096 Oct 13 09:07 github
drwxr-xr-x    1 root   root       4096 Oct  9 12:02 home
-rw-r--r--    1 root   root   60559544 Oct  9 12:02 kubectl
lrwxrwxrwx    1 root   root          7 Apr 22  2024 lib -> usr/lib
lrwxrwxrwx    1 root   root          9 Apr 22  2024 lib64 -> usr/lib64
drwxr-xr-x    2 root   root       4096 Sep 25 02:07 media
drwxr-xr-x    2 root   root       4096 Sep 25 02:07 mnt
drwxr-xr-x    2 root   root       4096 Sep 25 02:07 opt
dr-xr-xr-x  203 root   root          0 Oct 13 09:07 proc
drwx------    2 root   root       4096 Sep 25 02:14 root
drwxr-xr-x    1 root   root       4096 Oct 13 09:07 run
lrwxrwxrwx    1 root   root          8 Apr 22  2024 sbin -> usr/sbin
drwxr-xr-x    2 root   root       4096 Sep 25 02:07 srv
dr-xr-xr-x   12 root   root          0 Oct 13 09:07 sys
drwxrwxrwt    2 root   root       4096 Sep 25 02:14 tmp
drwxr-xr-x    1 root   root       4096 Sep 25 02:07 usr
drwxr-xr-x    1 root   root       4096 Sep 25 02:14 var
drwxr-xr-x 6 runner runner 4096 Oct 13 09:07 /__w

Below is the output when using our custom runner.
You can see that the __w directory is owned by root while the workflow pod is running with uid 1001 resulting in the permission denied error.

runner@default-staging-24m5f-runner-xr4z9-workflow:/__w/tm-pnt-dummy/tm-pnt-dummy$ ls -ltr /
total 56836
lrwxrwxrwx   1 root root        8 Apr 22  2024 sbin -> usr/sbin
lrwxrwxrwx   1 root root        7 Apr 22  2024 lib -> usr/lib
drwxr-xr-x   2 root root     4096 Apr 22  2024 boot
lrwxrwxrwx   1 root root        7 Apr 22  2024 bin -> usr/bin
drwxr-xr-x   1 root root     4096 Sep 25 02:11 usr
drwxr-xr-x   2 root root     4096 Sep 25 02:11 srv
drwxr-xr-x   2 root root     4096 Sep 25 02:11 opt
drwxr-xr-x   2 root root     4096 Sep 25 02:11 mnt
drwxr-xr-x   2 root root     4096 Sep 25 02:11 media
drwxr-xr-x   1 root root     4096 Sep 25 02:17 var
drwx------   2 root root     4096 Sep 25 02:17 root
drwxrwxrwt   2 root root     4096 Sep 25 02:17 tmp
-rw-r--r--   1 root root 58130616 Oct  9 12:02 kubectl
drwxr-xr-x   1 root root     4096 Oct  9 12:02 home
drwxr-xr-x   1 root root     4096 Oct  9 12:02 etc
dr-xr-xr-x  12 root root        0 Oct 13 08:13 sys
drwxrwxrwx   2 root root     4096 Oct 13 09:17 github
drwxrwxrwx   4 root root     4096 Oct 13 09:17 __e
dr-xr-xr-x 283 root root        0 Oct 13 09:17 proc
drwxr-xr-x   1 root root     4096 Oct 13 09:17 run
drwxr-xr-x   5 root root      360 Oct 13 09:17 dev
drwxr-xr-x   3 root root     4096 Oct 13 09:17 __w

runner@default-staging-24m5f-runner-xr4z9-workflow:/__w/tm-pnt-dummy/tm-pnt-dummy$ id -u
1001

runner@default-staging-24m5f-runner-xr4z9-workflow:/__w/tm-pnt-dummy/tm-pnt-dummy$ id -g
1001

Do you have any idea where the /__w directory is created and why this is owned by root?

Thanks in advance!

[vvanouytsel]

I have a feeling this bug could be reproducible as long as you use a container block that uses a container that is not running as root.

[marcusramberg]

We randomly started seeing this issue on our arm runners (with images running as 1001) after upgrading to 0.8.0 - Had to revert back to 0.7.0. It's also not very helpful that the reported error message is {}.

[vvanouytsel]

@nikola-jokic , where you able to reproduce this by any chance?
It should be reproducible as long as you run it in a container that is not running as root.

  test-custom-runner-with-container:
    runs-on: my-custom-runner
    container:
      image: my-company.registry.com/my-container:1.2.3 # this should not run as root
    steps:
      - name: List
        run: |
          id -u
          id -g
          ls -l /
          ls -ld /__w

##[debug]cpToPod: Attempt 1 failed: Error: Error from cpFromPod - details: 
##[debug] tar: _PipelineMapping: Cannot mkdir: Permission denied
##[debug]tar: _temp: Cannot mkdir: Permission denied
##[debug]tar: _tool: Cannot mkdir: Permission denied

[wd-hopkins]

I'm seeing this as well. In our case we have a hard requirement that pods do not run as root, so this is blocking us from utilising the new version