helm.v4.Chart: allow autonaming resource (allows graceful statefulset replacement)

[awoimbee]

Hi,
When I update statefulsets that are part of helm charts, I get errors like:

error: Preview failed: resource "urn:pulumi:[...]" was not successfully created by the Kubernetes API server: StatefulSet.apps "valkey-node" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Both at preview and at apply time.

The issue is: pulumi wants to recreate the statefulset, but it's using the same name, and the SSA API doesn't differentiate update from creation.
I could delete-before-replace, but that would cause downtime.
I would like a way to autoname this statefulset so pulumi can create-then-delete.
Note that I just want to create-then-delete the statefulset, I don't want to change the name of the whole chart.

I tried:

• Setting "pulumi.com/autonamed": didn't work as it's an annotation that pulumi creates but doesn't read to know if something should be autonamed
• props.metadata.name = undefined;: triggers autonaming, but:
  • I have no control over the given name
  • The generated name is invalid: valkey:base/valkey-node-094adbbd
• A random name suffix:

  transforms: [({opts, props}) => {
    if (props.kind === "StatefulSet") {
      const randomStr = Math.random().toString(36).substring(2, 6);
      props.metadata.name = `${props.metadata.name}-${randomStr}`;
      return { opts, props };
    }
    return undefined;
  }]

  The issue is that this would always cause a change and recreate the sts.
• Using metadata.generateName: doesn't work as SSA doesn't differentiate create and update

Proposed solution:
autonaming for chart resources strip the prefix (valkey:base/valkey-node-094adbbd -> valkey-node-094adbbd)

Issue with this solution: while it unlocks advanced users, it's terrible for newcomers (you need to use a transform and set props.metadata.name = undefined;...)

[awoimbee]

Wait what I saw this in the code, but it doesn't work for statefulsets as they don't return FieldImmutableErrorMsg

pulumi-kubernetes/provider/pkg/await/await.go

Line 200 in 04cc31b

// Handle the preview scenario where we need to re-create the object due to immutable fields.

And why is pulumi-kubernetes only using Patch and never Create in the first place ?

[blampe]

The issue is that this would always cause a change and recreate the sts.

Can you use the random provider?

And why is pulumi-kubernetes only using Patch and never Create in the first place ?

A very reasonable question #2998

[awoimbee]

Can you use the random provider?

So I tried this:

transforms: [
        ({ props, opts }) => {
          if (props.kind === "StatefulSet") {
            const id = new RandomString("valkey-statefulset-id", { length: 4, keepers: props }).result;
            props.metadata.name = id.apply(i => `${props.metadata.name}-${i}`);
            return { props, opts };
          }
          return undefined;
        },
      ],

But pulumi just hangs. I guess it's because I'm introducing a circular dep between the statefulset and the RandomString ?

So I tried

const id = new RandomString("valkey-statefulset-id", { length: 4, keepers: { ...props, metadata: { ...props.metadata } } }).result;

But I get

  random:index:RandomString (valkey-statefulset-id):
    error: cannot encode resource inputs to call ValidateResourceConfig: objectEncoder failed on property "keepers": encMap failed on {map[labels:{map[app.kubernetes.io/component:{node} app.kubernetes.io/instance:{valkey} app.kubernetes.io/managed-by:{Helm} app.kubernetes.io/name:{valkey} app.kubernetes.io/part-of:{valkey} app.kubernetes.io/version:{8.1.2} helm.sh/chart:{valkey-3.0.16}]} name:{valkey-node} namespace:{base}]}

If I can't get automatic keepers I might as well just manually write '-bis', '-ter', '-quater', ...

[blampe]

Totally agree the transform workaround is awkward and non-obvious.

The generated name is invalid: valkey:base/valkey-node-094adbbd
Proposed solution:
autonaming for chart resources strip the prefix (valkey:base/valkey-node-094adbbd -> valkey-node-094adbbd)

This seems reasonable although I thought this naming was getting offloaded to helm. I could be wrong. To confirm, this is a v4 chart resource you're working with?

@awoimbee could you include the relevant helm chart and some values that reproduce the issue? That will give us a repro. I'm also curious how things end up getting named if you install the same chart/values via the helm CLI directly.

[EronWright]

One suggestion that I have seen is to use {{ .Release.Revision }} in the chart template, to create a new object on each release. However, this would not work with the Chart resource because it doesn't create or use Helm releases. This might be a case for using the Release resource.

Stiil, the concept of side-by-side StatefulSets strikes me as highly unusual, and I would really strive to update the StatefulSet in compatible ways.

Regarding support for generateName, it is generally not supported by any of the provider's resources that work directly with Kubernetes manifests (Chart, ConfigGroup, etc). See also: helm/helm#3348

Note that auto-naming still relies on you giving the resource a logical name. The above-mentioned resource types derive the resource name from the object name; we'd need a new way to name them in order to support auto-naming.

[awoimbee]

One suggestion that I have seen is to use {{ .Release.Revision }} [...]

Creating a new statefulset for each revision seems overkill

Stiil, the concept of side-by-side StatefulSets strikes me as highly unusual [...]

True, it doesn't happen every day.

Note that auto-naming still relies on you giving the resource a logical name.

In my case, pulumi was naming the statefulset valkey:base/valkey-node-094adbbd which is {chartName}:{namespace}/{statefulsetName}-{uid}. Thinking about it, it's weird that it got that statefulset name since I set it to undefined in my transform. I think it's because resource registration happens before transforms.
TL;DR: My PR is a hack but it really works

---

All in all the best solution would be to delegate this to k8s using generateName (#2998).