chore(ci): add static config for deploy nested cluster and configure ci for nightly e2e tests #1876

universal-itengineer · 2026-01-14T21:00:54Z

Description

add static config for deploy nested cluster and configure ci for nightly e2e tests

Why do we need it, and what problem does it solve?

What is the expected result?

Checklist

The code is covered by unit tests.
e2e tests passed.
Documentation updated according to the changes.
Changes were tested in the Kubernetes cluster manually.

Changelog entries

section:
type:
summary:

fl64

I suggest check all the sh scripts via https://www.shellcheck.net and fix all the issues.

Signed-off-by: Nikita Korolev <[email protected]>

diafour · 2026-01-16T13:05:57Z

.github/workflows/e2e-reusable-pipeline.yml

        required: false
        type: string
-        default: "1.24.5"
+        default: "1.24.6"


"golang-alt-1.24" base image uses 1.24.11

For e2e tests we use golang v1.24.6

.github/workflows/e2e-reusable-pipeline.yml

diafour · 2026-01-16T13:19:59Z

.github/workflows/e2e-reusable-pipeline.yml

+          echo "login to master"
+          echo "os-release master"
+          d8vssh 'cat /etc/os-release'
+          echo ""
+
+          echo "hostname master"
+          d8vssh 'hostname'


Why 2 connections?

Suggested change

echo "login to master"

echo "os-release master"

d8vssh 'cat /etc/os-release'

echo ""

echo "hostname master"

d8vssh 'hostname'

echo "Check connection to master"

d8vssh 'echo master os-release: ; cat /etc/os-release; echo master hostname: ; hostname'

echo ""

diafour · 2026-01-16T13:23:31Z

.github/workflows/e2e-reusable-pipeline.yml

+          CONFIG=$(cat ${kubeConfigPath} | base64 -w 0)
+          CONFIG=$(echo $CONFIG | base64 -w 0)


Why 2 commands?

Suggested change

CONFIG=$(cat ${kubeConfigPath} | base64 -w 0)

CONFIG=$(echo $CONFIG | base64 -w 0)

CONFIG=$(cat ${kubeConfigPath} | base64 -w 0 | base64 -w 0)

diafour · 2026-01-16T13:24:25Z

.github/workflows/e2e-reusable-pipeline.yml

+          echo "[INFO] Kubeconf to github output"
+          CONFIG=$(cat ${kubeConfigPath} | base64 -w 0)
+          CONFIG=$(echo $CONFIG | base64 -w 0)
+          echo "config=$CONFIG" >> $GITHUB_OUTPUT


config seems too broad, may be name it kubeconfig ?

diafour · 2026-01-16T13:27:20Z

.github/workflows/e2e-reusable-pipeline.yml

+        run: |
+          sudo chown -fR 1001:1001 ${{ env.SETUP_CLUSTER_TYPE_PATH }}
+          yq e '.deckhouse.registryDockerCfg = "None"' -i ./${{ env.SETUP_CLUSTER_TYPE_PATH }}/values.yaml
+          yq e 'select(.kind == "InitConfiguration") .deckhouse.registryDockerCfg = "None"' -i ./${{ env.SETUP_CLUSTER_TYPE_PATH }}/tmp/config.yaml || echo "The config.yaml file is not generated, skipping"


Is yq expression ok? It seems | is missing after select.

.github/workflows/e2e-reusable-pipeline.yml

diafour · 2026-01-16T13:31:29Z

.github/workflows/e2e-reusable-pipeline.yml

+      kubeconfig-content: ${{ steps.generate-kubeconfig.outputs.config }}
+      storage-type: ${{ steps.vars.outputs.storage_type }}
+      nested-storageclass-name: ${{ steps.vars.outputs.nested_storageclass_name }}


Please, make steps output names equal to job output names.

diafour · 2026-01-16T13:34:38Z

.github/workflows/e2e-reusable-pipeline.yml

+          echo "[INFO] Show nodes in cluster"
+          kubectl config get-contexts
+
+          # some times kubectl get nodes returns error, so we need to retry


Suggested change

# some times kubectl get nodes returns error, so we need to retry

# `kubectl get nodes` may return error, so we need to retry.

diafour · 2026-01-16T13:38:10Z

.github/workflows/e2e-reusable-pipeline.yml

+
+  configure-virtualization:
+    name: Configure Virtualization (${{ inputs.storage_type }})
+    runs-on: ubuntu-22.04


Workflow has different runs-on values (bootstrap and other configure jobs use ubuntu-latest, this one uses ubuntu-22.04). Why?

diafour · 2026-01-16T13:47:54Z

.github/workflows/e2e-reusable-pipeline.yml

+          STORAGE_CLASS_NAME: ${{ inputs.nested_storageclass_name }}
+        working-directory: ./test/e2e/
+        run: |
+          GINKGO_RESULT=$(mktemp)


There is a temp directory for each run, no need to create one, just use $RUNNER_TEMP

diafour · 2026-01-16T13:54:05Z

test/dvp-static-cluster/Taskfile.yaml

+        echo vms:
+        kubectl -n {{ .NAMESPACE }} get vm
+        echo "Grafana URL https://grafana.{{ .NAMESPACE }}.{{ .DOMAIN }}"
+        echo "Default user/password [email protected]/{{ .PASSWORD}}"


Seems not secure

test/dvp-static-cluster/scripts/gen-kubeconfig.sh

test/dvp-static-cluster/storage/ceph/03-cm.yaml

test/dvp-static-cluster/storage/ceph/05-blockpool.yaml

diafour · 2026-01-16T14:52:00Z

test/dvp-static-cluster/storage/sds-replicated/rsc-gen.sh

+kubectl apply -f ${manifest}
+
+DEFAULT_STORAGE_CLASS=nested-thin-r1
+kubectl patch mc global --type='json' -p='[{"op": "replace", "path": "/spec/settings/defaultClusterStorageClass", "value": "'"$DEFAULT_STORAGE_CLASS"'"}]'


Also, add all other SC to /spec/settings/allowedStorageClasses array. (StorageClassChange test will fail is defaultClusterStorageClass is set without allowedStorageClasses)

diafour · 2026-01-16T14:59:29Z

test/dvp-static-cluster/charts/infra/templates/_helpers.tpl

+      users:
+        - default
+        - name: cloud
+          passwd: $6$rounds=4096$vln/.aPHBOI7BMYR$bBMkqQvuGs5Gyd/1H5DP4m9HjQSy.kgrxpaGEHwkX7KEFV8BS.HZWPitAtZ2Vd8ZqIZRqmlykRCagTgPejt1i.


Is it possible to store this password in github secrets?

test/dvp-static-cluster/charts/infra/templates/jump-host/deploy.yaml

test/dvp-static-cluster/charts/cluster-config/templates/modules-dvp-base.yaml

Signed-off-by: Nikita Korolev <[email protected]>

github-actions bot assigned universal-itengineer Jan 14, 2026

universal-itengineer requested a review from diafour January 14, 2026 21:20

fl64 reviewed Jan 15, 2026

View reviewed changes

universal-itengineer force-pushed the feat/ci/nightly-e2e-test-infra branch from d40af14 to 6908b14 Compare January 15, 2026 13:03

universal-itengineer requested a review from fl64 January 15, 2026 13:09

universal-itengineer added 3 commits January 15, 2026 19:37

chore(ci): add ci for deploy nested cluster for nightly e2e tests

50e44e0

Signed-off-by: Nikita Korolev <[email protected]>

chore(ci): add static config for nested cluster

39d5e3a

Signed-off-by: Nikita Korolev <[email protected]>

use release-1.74 for deckhouse deploy

247ca1f

Signed-off-by: Nikita Korolev <[email protected]>

universal-itengineer force-pushed the feat/ci/nightly-e2e-test-infra branch from 6908b14 to 247ca1f Compare January 15, 2026 16:37

universal-itengineer added 3 commits January 15, 2026 21:00

comment proxy settings

cb1af7a

Signed-off-by: Nikita Korolev <[email protected]>

hardcode d8 cli version to v0.25.1

2546b79

Signed-off-by: Nikita Korolev <[email protected]>

add Second check d8 queue in nested cluster

3b8a7c1

Signed-off-by: Nikita Korolev <[email protected]>

universal-itengineer force-pushed the feat/ci/nightly-e2e-test-infra branch from be94996 to 3b8a7c1 Compare January 16, 2026 10:08

universal-itengineer added 2 commits January 16, 2026 14:22

upd wait sds-replicated

ff0dead

Signed-off-by: Nikita Korolev <[email protected]>

upd wait nested kube-api

1d82a46

Signed-off-by: Nikita Korolev <[email protected]>

universal-itengineer force-pushed the feat/ci/nightly-e2e-test-infra branch from 97b4b99 to 1d82a46 Compare January 16, 2026 12:35

diafour reviewed Jan 16, 2026

View reviewed changes

.github/workflows/e2e-reusable-pipeline.yml Outdated Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

.github/workflows/e2e-reusable-pipeline.yml Outdated Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

.github/workflows/e2e-reusable-pipeline.yml Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

.github/workflows/e2e-reusable-pipeline.yml Outdated Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

test/dvp-static-cluster/scripts/gen-kubeconfig.sh Outdated Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

test/dvp-static-cluster/storage/ceph/03-cm.yaml Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

test/dvp-static-cluster/storage/ceph/05-blockpool.yaml Outdated Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

test/dvp-static-cluster/charts/infra/templates/jump-host/deploy.yaml Outdated Show resolved Hide resolved

diafour reviewed Jan 16, 2026

View reviewed changes

test/dvp-static-cluster/charts/cluster-config/templates/modules-dvp-base.yaml Outdated Show resolved Hide resolved

resolve comments

ffa3cd8

Signed-off-by: Nikita Korolev <[email protected]>

		CONFIG=$(cat ${kubeConfigPath} \| base64 -w 0)
		CONFIG=$(echo $CONFIG \| base64 -w 0)

	# some times kubectl get nodes returns error, so we need to retry
	# `kubectl get nodes` may return error, so we need to retry.

chore(ci): add static config for deploy nested cluster and configure ci for nightly e2e tests #1876

Are you sure you want to change the base?

chore(ci): add static config for deploy nested cluster and configure ci for nightly e2e tests #1876

Uh oh!

Conversation

universal-itengineer commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Why do we need it, and what problem does it solve?

What is the expected result?

Checklist

Changelog entries

Uh oh!

fl64 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diafour Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

diafour Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

universal-itengineer commented Jan 14, 2026 •

edited

Loading

fl64 left a comment •

edited

Loading

diafour Jan 16, 2026 •

edited

Loading

diafour Jan 16, 2026 •

edited

Loading