Support for efficient stack configuration updates

[JonCholas]

Hello!

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

In the current design, the stack controller populates the stack field in the workspace CRD. The stacks are composed of the config and some other stack configurations.

When the workspace starts (aka is ready), the workspace_controller sets the stack configuration based on the workspace stack field

if len(stack.Config) != 0 {
	l.V(1).Info("Setting the stack configuration")
	config := make([]*agentpb.ConfigItem, 0, len(stack.Config))
	for _, item := range stack.Config {
		config = append(config, marshalConfigItem(item))
	}
	_, err := wc.SetAllConfig(ctx, &agentpb.SetAllConfigRequest{
		Config: config,
	})
	if err != nil {
		return err
	}
}

This creates a 1:1 dependency between the Workspace and the stack configuration. So every time the stack config changes, the stack generation changes, and then the workspace generation changes and therefore the workspace gets a rollout deployment (stateful changes).

A workspace is meant to be a runtime for the stack and should be independent of the stack config and any other field that doesn't affect the workspace pod.

The current methods waste a ton of resources re-initialising the pod (not to mention re-scheduling a new one, which will depend on the autoscaler to provision a new node or be able to re-utilise the current one) when the pod doesn't need to change, as the stack config is independent of the runtime environment.

Other stack changes should affect the pod, like the podTemplate or other arguments that are injected into the workspace (like service_account), but nothing else.

Is there any reason why the stack config shouldn't be on the update CRD instead? It would make more sense to have the stack config per update to track different configs on each update.

Affected area/feature

Pulumi operator

[EronWright]

It is true that the pod is tied to a stack, and is replaced when/if the workspace's spec is changed.

The main reason for the current design is to maintain a hermetic environment. The pulumi config set command works by mutating the local stack configuration file, and is additive to what was in source control. Let's say you changed the stack config of a given Workspace object, such that a particular setting was removed (say, 'replicas'). The correct behavior is for X to revert to the value that is in source control (if any). This is accomplished by replacing the pod whenever the specification or the source code is updated.

Still, it would be nice to optimize this case.

[JonCholas]

It is true that the pod is tied to a stack, and is replaced when/if the workspace's spec is changed.

The main reason for the current design is to maintain a hermetic environment. The pulumi config set command works by mutating the local stack configuration file, and is additive to what was in source control. Let's say you changed the stack config of a given Workspace object, such that a particular setting was removed (say, 'replicas'). The correct behavior is for X to revert to the value that is in source control (if any). This is accomplished by replacing the pod whenever the specification or the source code is updated.

Still, it would be nice to optimize this case.

This is why i think the stack config and the workspacePod template are different things that shoule be separated. Yes, changes on the podTemplate would affect the pod, but the stack config won't. I feel like the podTemplate should live somewhere else or there should be logic to detect when something is modified, for example, if the serviceAccountName is changed in the Stack spec then you need to re-evaluate the podTemplate because it will change, but if the stackConfig changes inside the Stack spec it shouldn't affect the workspace template at all.

Im not sure if the design should be "move the podTemplate" to another place or "add logic to detect changes" the same way pulumi detects "updates" in resources and sometimes the trigger a "replacement".

If you give me some guidance, I can give it a try to implement it. It is a feature that affects us greately because we use pulumi operator to do deployments and we do this by changing the image tag of our services. The image tag is part of the stack config and therefore on each deployment we need to wait for a replacement of the workspace pod, delaying the deployment by a fair bit (we need to wait for initialisation+git pull+ uv sync+pulumi to start when receives the call from the operator controller)

[EronWright]

Any change we make has to ensure that stack configuration is hermetic, i.e. that the actual stack configuration is exactly what is specified in the Stack spec plus the source-controlled configuration. If you mutate the stack configuration over time, one must 'revert' the stack configuration file (because Pulumi mutates it as a side-effect of config set), and replacing the pod is a uniform and effective way to do that. It does have the performance penalties that you mentioned, and I share your interest in optimizing it.

While we continue to discuss options, a workaround to consider is to use a Flux GitRepository source, which serves as a local cache of the repository. I find it most helpful for large repositories like pulumi/examples. Dependency installation is another pain point, which one may optimize by pre-installing dependencies into a custom image derived from pulumi/pulumi.