feat: Cache transform for Secrets and ConfigMaps to reduce memory #1866

svghadi · 2025-09-11T07:25:37Z

What type of PR is this?
/kind enhancement

What does this PR do / why we need it:

This PR reduces memory usage in the Argo CD Operator by introducing a cache transform and client wrapper for Secrets and ConfigMaps.

Cache transform: Strips large data fields from non-operator Secrets/ConfigMaps before storing them in the controller-runtime cache, while retaining full content for operator-tracked objects.
Client wrapper: Falls back to the live client when a stripped object is accessed and applies a tracking label so future updates are fully cached.
Integration: Wired into main.go via cache.Options.ByObject and applied across reconcilers.

See the attached proposal in this PR for design details, trade-offs, PoC results, and future scope.

proposal

Results
These metrics were collected from a test cluster containing 100 ConfigMaps and 100 Secrets, each approximately 1 MB in size. The cluster was running four Argo CD instances and no other workload operators.

With the optimization enabled, operator memory usage dropped from ~350 MB to ~100 MB.

UnOptimized Operator Manager Memory:

Optimized Operator Manager Memory:

However, we could not reduce the startup memory consumption, which remained at ~750 MB in both cases.

We previously attempted another approach in #1795, but it introduced significant complexity and restricted how watches could be set up. Compared to that, this solution provides a better balance between complexity, maintenance overhead, and outcome.

Signed-off-by: Siddhesh Ghadi <[email protected]>

anandf · 2025-09-23T04:57:08Z

pkg/cacheutils/utlis.go

+				return in, nil
+			}
+			// Strip data for non-operator secrets
+			return &v1.Secret{


Rather than creating a new Object, can we not set the Data field to nil on the same object ? When we create new Secret object. The old object with the data field also exists in memory and waits for Garbage collection to clean it up, isn't it ?

Hey @anandf I have updated it according to how you have mentioned in comment

anandf · 2025-09-23T04:59:50Z

pkg/cacheutils/utlis.go

+// StripConfigMapDataTransform returns a TransformFunc that strips the data from ConfigMaps
+// that are not tracked by the operator. This is useful for reducing memory usage
+// when caching ConfigMaps that are not managed by the operator.
+func StripConfigMapDataTransform() clientgotools.TransformFunc {


Can we have a single generic function that can handle both Secret and ConfigMap ? Anyways the function takes in an generic interface{} struct.

Unified both function

anandf · 2025-09-23T05:01:11Z

pkg/cacheutils/utlis.go

+}
+
+// IsTrackedByOperator checks if the given labels indicate that the resource is tracked by the operator.
+func IsTrackedByOperator(labels map[string]string) bool {


Can it take in a runtime.Object as input instead of labels ? That would be more meaningful as the method checks if a given resource is tracked by operator or not.

@anandrkskd - can we incorporate this suggestion aswell?

Signed-off-by: Anand Kumar Singh <[email protected]>

svghadi · 2025-10-23T05:19:58Z

pkg/clientwrapper/client.go

Is it possible to add some tests for the wrapper?

svghadi · 2025-10-23T05:24:57Z

docs/usage/environment_variables.md

 | `REMOVE_MANAGED_BY_LABEL_ON_ARGOCD_DELETION` | false | When an Argo CD instance is deleted, namespaces managed by that instance (via the `argocd.argoproj.io/managed-by` label ) will retain the label by default. Users can change this behavior by setting the environment variable `REMOVE_MANAGED_BY_LABEL_ON_ARGOCD_DELETION` to `true` in the Subscription. |
 | `ARGOCD_LABEL_SELECTOR` | none | The label selector can be set on argocd-opertor by exporting `ARGOCD_LABEL_SELECTOR` (eg: `export ARGOCD_LABEL_SELECTOR=foo=bar`). The labels can be added to the argocd instances using the command `kubectl label argocd test1 foo=bar -n test-argocd`. This will enable the operator instance to be tailored to oversee only the corresponding ArgoCD instances having the matching label selector. |
 | `LOG_LEVEL` | info | This sets the logging level of the manager (operator) pod. Valid values are "debug", "info", "warn", "error", "panic" and "fatal". |
+| `MEMORY_OPTIMIZATION_ENABLED` | true | Enables memory optimization by stripping data from Secrets and ConfigMaps that are not tracked by the operator, reducing memory usage. Set to `false` to disable this optimization. |


Suggested change

| `MEMORY_OPTIMIZATION_ENABLED` | true | Enables memory optimization by stripping data from Secrets and ConfigMaps that are not tracked by the operator, reducing memory usage. Set to `false` to disable this optimization. |

| `DISABLE_MEMORY_OPTIMIZATION` | false | When set to `true`, disables the memory optimization that strips data from Secrets and ConfigMaps that are not tracked by the operator. This optimization helps reduce memory usage. |

svghadi · 2025-10-23T05:33:51Z

cmd/main.go

-	if watchedNsCache := getDefaultWatchedNamespacesCacheOptions(); watchedNsCache != nil {
+	// Use transformers to strip data from Secrets and ConfigMaps
+	// that are not tracked by the operator to reduce memory usage.
+	if strings.ToLower(os.Getenv("MEMORY_OPTIMIZATION_ENABLED")) != "false" {


Lets use DISABLE_MEMORY_OPTIMIZATION env var which users should explicitly set to true to disable optimization

svghadi · 2025-10-23T05:36:26Z

cmd/main.go

+	liveClient, err := crclient.New(ctrl.GetConfigOrDie(), crclient.Options{Scheme: mgr.GetScheme()})
+	if err != nil {
+		setupLog.Error(err, "unable to create live client")
+		os.Exit(1)
+	}
+
+	// Wraps the controller runtime's default client to provide:
+	//   1. Fallback to the live client when a Secret/ConfigMap is stripped in the cache.
+	//   2. Automatic labeling of fetched objects, so they are retained in full form
+	//      in subsequent cache updates and avoid repeated live lookups.
+	client := cw.NewClientWrapper(mgr.GetClient(), liveClient)
+


This logic can also be placed behind the DISABLE_MEMORY_OPTIMIZATION check. Use the default client := mgr.GetClient() when memory optimization is disabled.

svghadi marked this pull request as draft September 11, 2025 07:25

svghadi force-pushed the mem-optmz branch from e093189 to f1adb78 Compare September 11, 2025 10:04

anandrkskd mentioned this pull request Sep 22, 2025

Downstream memory optimization redhat-developer/gitops-operator#972

Draft

4 tasks

svghadi changed the title ~~WIP~~ feat: Cache transform for Secrets and ConfigMaps to reduce memory Sep 22, 2025

svghadi added 3 commits September 22, 2025 22:09

WIP

fefdf26

Signed-off-by: Siddhesh Ghadi <[email protected]>

Code clenup and commenting

15856bf

Signed-off-by: Siddhesh Ghadi <[email protected]>

Add proposal doc

013277d

Signed-off-by: Siddhesh Ghadi <[email protected]>

svghadi force-pushed the mem-optmz branch from 4224175 to 013277d Compare September 22, 2025 16:42

svghadi marked this pull request as ready for review September 22, 2025 16:45

anandf reviewed Sep 23, 2025

View reviewed changes

anandrkskd added 3 commits October 13, 2025 16:57

merge transform func, add tests for configmap and secret

40c088d

Signed-off-by: Anand Kumar Singh <[email protected]>

add env var to disable optimization, update docs

3a70ac4

Signed-off-by: Anand Kumar Singh <[email protected]>

fix CI lint failure

06fba6f

Signed-off-by: Anand Kumar Singh <[email protected]>

svghadi commented Oct 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: Cache transform for Secrets and ConfigMaps to reduce memory #1866

feat: Cache transform for Secrets and ConfigMaps to reduce memory #1866

Uh oh!

svghadi commented Sep 11, 2025 •

edited

Loading

Uh oh!

anandf Sep 23, 2025

Uh oh!

anandrkskd Oct 16, 2025

Uh oh!

anandf Sep 23, 2025

Uh oh!

anandrkskd Oct 16, 2025 •

edited

Loading

Uh oh!

anandf Sep 23, 2025

Uh oh!

svghadi Oct 23, 2025

Uh oh!

svghadi Oct 23, 2025

Uh oh!

svghadi Oct 23, 2025

Uh oh!

svghadi Oct 23, 2025

Uh oh!

svghadi Oct 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	\| `MEMORY_OPTIMIZATION_ENABLED` \| true \| Enables memory optimization by stripping data from Secrets and ConfigMaps that are not tracked by the operator, reducing memory usage. Set to `false` to disable this optimization. \|
	\| `DISABLE_MEMORY_OPTIMIZATION` \| false \| When set to `true`, disables the memory optimization that strips data from Secrets and ConfigMaps that are not tracked by the operator. This optimization helps reduce memory usage. \|

Uh oh!

feat: Cache transform for Secrets and ConfigMaps to reduce memory #1866

Are you sure you want to change the base?

feat: Cache transform for Secrets and ConfigMaps to reduce memory #1866

Uh oh!

Conversation

svghadi commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anandrkskd Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

svghadi commented Sep 11, 2025 •

edited

Loading

anandrkskd Oct 16, 2025 •

edited

Loading