PodLogs pod discovery change

[talpa-robin]

Hello everybody,

recently we stumbled upon a problem where the alloy-logs pods are running OOM after some time, but only for nodes with specific pods on them. That's odd, so what happened?

1. We have a StatefulSet running where the pods include annotations with some kind of state (it really doesn't matter but it's a Postgres database with Patroni for clustering).
2. The value of an annotation with the state changes every 10s.
3. Because of the annotation change the target in Alloy changes / resets.
4. The resetting of the target leads to the offset in the positions.yaml to be reset.
5. Alloy starts ingesting the same pod logfile over and over again.
6. This, after some time, leads to log pressure and the alloy-logs pod going OOM.

So while this is a behavior of Alloy (and we could debate if that is working as intended), this chart has some configuration that makes it possible to happen in the first place.

I'm talking about this template snippet

k8s-monitoring-helm/charts/k8s-monitoring/charts/feature-pod-logs/templates/_common_pod_discovery.alloy.tpl

Lines 52 to 55 in b0e967a

	rule {
	action = "labelmap"
	regex = "__meta_kubernetes_pod_annotation_(.+)"
	}

With that configuration all annotations are made available as labels to Alloy and therefore a change of the annotations create a new target which then leads to the described behavior.

We now explicitly drop the affected annotations / labels and that fixes the problem, but that is only a solution for this one workload / annotation. In the future there might be other workloads which show the same behavior with a different annotation.

My questions now would be:

1. Is the mentioned code, or the result of it (having the annotations as labels in the pipelines) used somewhere else or what was the intention for doing it in the first place?
2. Would it be possible to make that part configurable so that we can either turn the annotations -> label function on / off or provide a set of annotations which should be made accessible within the labels?

Thanks for reading, I'm curious about your opinions.