OKD-294: Migrate runtime from runc to crun on an upgrade for OKD #5389

Prashanth684 · 2025-11-03T23:03:29Z

Centos stream 10 dopped the runc package. For now OKD users are following the workaround to edit the MC to point to crun before the upgrade, but this should help with the migration rather than having to intervene manually.

This PR:

checks for presence of the configmap
if MCO is built for OKD, edit the MC so the runtime is crun.
deletes the configmap

Borrows from the implementation here: #4635

openshift-ci-robot · 2025-11-03T23:03:33Z

@Prashanth684: This pull request references OKD-294 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

Centos stream 10 dopped the runc package. For now OKD users are following the workaround to edit the MC to point to crun before the upgrade, but this should help with the migration rather than having to intervene manually.

This PR:

checks for presence of the configmap

if MCO is built for OKD, edit the MC so the runtime is crun.

deletes the configmap

Borrows from the mplementation here: #4635

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-11-03T23:03:43Z

@Prashanth684: This pull request references OKD-294 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

Centos stream 10 dopped the runc package. For now OKD users are following the workaround to edit the MC to point to crun before the upgrade, but this should help with the migration rather than having to intervene manually.

This PR:

checks for presence of the configmap

if MCO is built for OKD, edit the MC so the runtime is crun.

deletes the configmap

Borrows from the implementation here: #4635

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-11-03T23:04:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Prashanth684
Once this PR has been reviewed and has the lgtm label, please assign kolyshkin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/controller/container-runtime-config/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Prashanth684 · 2025-11-04T17:45:17Z

/test bootstrap-unit

Prashanth684 · 2025-11-04T17:45:24Z

/test okd-scos-e2e-aws-ovn

Centos stream 10 dopped the runc package. For now OKD users are following the workaround to edit the MC to point to crun before the upgrade, but this should help with the migration rather than having to intervene manually. This PR: - checks for presence of the configmap - if MCO is built for OKD, edit the MC so the runtime is crun. - deletes the configmap

isabella-janssen · 2025-11-06T10:40:26Z

/retest-required

isabella-janssen · 2025-11-06T10:53:19Z

pkg/controller/container-runtime-config/container_runtime_config_controller.go

+	// Only process master and worker pools for the migration
+	for _, pool := range pools {
+		if pool.Name != ctrlcommon.MachineConfigPoolMaster && pool.Name != ctrlcommon.MachineConfigPoolWorker {
+			continue
+		}


From the context in #4635 (comment), I'm gathering that the intention is to only target default MCPs. What about the arbiter MCP? Should it also be included in this list of default MCPs?

machine-config-operator/pkg/controller/common/constants.go

Lines 66 to 67 in a1ac217

// MachineConfigPoolArbiter is the MachineConfigPool name given to the arbiter

MachineConfigPoolArbiter = "arbiter"

cc @yuqi-zhang Since it looks like you had the original suggestion to scope to default MCPs.

@yuqi-zhang do we need to consider the arbiter here? or are we good with the changes as is?

I think arbiter probably isn't relevant here, since 2-node-with-fencing is not yet GA'ed and even if it was, I don't think there would be any OKD overlap with it atm for upgrades, so we should be good.

So basically this is "forcing" any OKD clusters to use crun since runc is gone on upgrades, is installs still possible for runc if you set it (and is there a guard against it if so, I forget if we deprecated it)? Is RHEL10 also dropping runc? Maybe we need this generally speaking cc @sdodson

Yes, RHEL10 dropped runc but AFAIK the plan is to start building it for OCP as we're hesitant to drop runc until crun sees more adoption even if crun is the default since 4.18.

openshift-ci · 2025-11-06T12:26:56Z

@Prashanth684: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/bootstrap-unit	`7700e81`	link	false	`/test bootstrap-unit`
ci/prow/okd-scos-e2e-aws-ovn	`7700e81`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/e2e-hypershift	`7700e81`	link	true	`/test e2e-hypershift`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

yuqi-zhang · 2025-11-12T23:57:47Z

pkg/controller/container-runtime-config/container_runtime_config_controller.go

+	// Only process master and worker pools for the migration
+	for _, pool := range pools {
+		if pool.Name != ctrlcommon.MachineConfigPoolMaster && pool.Name != ctrlcommon.MachineConfigPoolWorker {
+			continue
+		}


I think arbiter probably isn't relevant here, since 2-node-with-fencing is not yet GA'ed and even if it was, I don't think there would be any OKD overlap with it atm for upgrades, so we should be good.

So basically this is "forcing" any OKD clusters to use crun since runc is gone on upgrades, is installs still possible for runc if you set it (and is there a guard against it if so, I forget if we deprecated it)? Is RHEL10 also dropping runc? Maybe we need this generally speaking cc @sdodson

yuqi-zhang · 2025-11-13T00:05:35Z

pkg/controller/container-runtime-config/container_runtime_config_controller.go

+		}
+
+		// Get the MachineConfig name for this pool
+		mcName := fmt.Sprintf("00-override-%s-generated-crio-default-container-runtime", pool.Name)


Hmm, where is this MachineConfig coming from? If you're trying to override what #4635 introduced, that wrote to 99-%s-generated-crio-default-container-runtime I'm pretty sure. In our case instead of playing around with MCs and overrides, we should just remove any runc references in the MC and use system default I think? (which would be crun)

this is coming from #4715 which was a change introduced in 4.17 alone i believe.

Oh right, forgot that, would we be able to just delete 00-override-pool-generated-crio-default-container-runtime on behalf of the user? I was under the impression deleting the MC would reset to system defaults which would be crun in this case, and comes with the benefits of not having an additional MC left in the cluster, and we can remove this code block when we branch for 4.22 since all OKD clusters that would have gone through 4.21 would have removed the MC entirely.

(Unless there's a need to explicitly set this on OKD)

I was under the impression deleting the MC would reset to system

I thought the same as well...but folks in the OKD community seem to have tried that and it looks like deleting it wasn't enough to reset to crun. Let me double check that behavior.

Ack, editing the MC should be ok as well, just that it'll become unmanaged state if we were ever to want to change that in the future, so if we can get deleting to work that would probably be better.

yuqi-zhang · 2025-11-13T00:07:06Z

pkg/controller/container-runtime-config/container_runtime_config_controller.go


 	ctrl.clusterVersionLister = clusterVersionInformer.Lister()
 	ctrl.clusterVersionListerSynced = clusterVersionInformer.Informer().HasSynced
+	ctrl.queue.Add(forceSyncOnUpgrade)


Have you observed this to be necessary? The new controller is expected to re-gen all the MCs so it should always be synced post-new controller deployment

yes - i noticed that before i added this, the upgrade to an image with these changes did not change the MC at all.

yuqi-zhang · 2025-11-13T00:07:30Z

pkg/controller/container-runtime-config/container_runtime_config_controller.go

 		_, ok := cfg.GetAnnotations()[ctrlcommon.MCNameSuffixAnnotationKey]
 		arr := strings.Split(managedKey, "-")
-		// the first managed key value 99-poolname-generated-containerruntime does not have a suffix
+		// the first managed key value 00-override-poolname-generated-containerruntime does not have a suffix


Did we change this at some point?

#4715 was introduced in 4.17 which creates this mc

ack, but this comment specifically is referring to the containerruntimeconfig object-generated MCs, which comes with suffixes.

We don't generate multiple 00-override-poolname-generated-containerruntime anyways

ah..right...let me fix that up

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Nov 3, 2025

openshift-ci bot requested review from QiWang19 and mtrmac November 3, 2025 23:04

Prashanth684 force-pushed the mco-crun branch 2 times, most recently from 493b483 to 2c63371 Compare November 3, 2025 23:51

Prashanth684 force-pushed the mco-crun branch from 2c63371 to 7700e81 Compare November 4, 2025 19:25

isabella-janssen reviewed Nov 6, 2025

View reviewed changes

yuqi-zhang reviewed Nov 13, 2025

View reviewed changes

	// MachineConfigPoolArbiter is the MachineConfigPool name given to the arbiter
	MachineConfigPoolArbiter = "arbiter"

OKD-294: Migrate runtime from runc to crun on an upgrade for OKD #5389

Are you sure you want to change the base?

OKD-294: Migrate runtime from runc to crun on an upgrade for OKD #5389

Uh oh!

Conversation

Prashanth684 commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 3, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Nov 3, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Nov 3, 2025

Uh oh!

Prashanth684 commented Nov 4, 2025

Uh oh!

Prashanth684 commented Nov 4, 2025

Uh oh!

isabella-janssen commented Nov 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Nov 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Prashanth684 Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Prashanth684 commented Nov 3, 2025 •

edited

Loading

openshift-ci-robot commented Nov 3, 2025 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Nov 3, 2025 •

edited by openshift-ci bot

Loading

Prashanth684 Nov 13, 2025 •

edited

Loading