Grafana Operator installation fails due to readiness and liveness probe failing, no error log messages

[glowing-axolotl]

Describe the bug
In 2024, we tried installing the Grafana Operator on OpenShift through OperatorHub. Initially the installation failed and wasn't continued. Then recently we tried a new installation after cleaning all the related resources (configmaps, subscription, secret, CRDs etc...) but it still fails.

Version
Version is 5.19.4 by Grafana Labs through OpenShift's OperatorHub https://operatorhub.io/operator/grafana-operator/v5/grafana-operator.v5.19.4 .

To Reproduce
Steps to reproduce the behavior:

1. Go to 'OperatorHub' in an OpenShift cluster
2. Search for 'Grafana'
3. Install "Grafana Operator"
4. Select v5, installation on specific namespace (already tried also all namespaces various times, it will install in openshift-operators in that case) with Automatic upgrades.
5. Check the logs and events for the created deployment, "grafana-operator-controller-manager-v5".
6. The installation fails on CSV with "install failed: deployment grafana-operator-controller-manager-v5 not ready before timeout: deployment "grafana-operator-controller-manager-v5" exceeded its progress deadline".
7. This is due to the generated pod grafana-operator-controller-manager-v5-5595b667f6-f44tr never reaching a state of Ready 1/1 .

Expected behavior
The grafana-operator-controller-manager-v5 pod should reach a ready state. This works correctly in various labs and different environments. We couldn't reproduce this bug in any other environments. For some unexpected reason the issue only occurs on this specific cluster.

Suspect component/Location where the bug might be occurring
The events for the pod show the probes failing:

Readiness probe error: Get "http://10.247.7.91:8081/readyz": read tcp 10.247.6.2:39806->10.247.7.91:8081: read: connection reset by peer body:

Liveness probe failed: Get "http://10.247.7.91:8081/healthz": EOF

Logs don't show any particular errors (the SIGTERM received most probably comes from the failing probe):

2025-09-12T14:19:38.879Z	info	setup	GOMEMLIMIT is updated	{"version": "v5.19.4", "package": "github.com/KimMachineGun/automemlimit/memlimit", "GOMEMLIMIT": 519045120, "previous": 9223372036854775807}
2025-09-12T14:19:38.880Z	info	setup	maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS	{"version": "v5.19.4"}
2025-09-12T14:19:38.977Z	info	setup	label restrictions for cached resources are active	{"version": "v5.19.4", "level": "safe"}
2025-09-12T14:19:38.999Z	info	setup	operator running in namespace scoped mode	{"version": "v5.19.4", "namespace": "test-monitoring"}
2025-09-12T14:19:39.064Z	info	setup	starting manager	{"version": "v5.19.4"}
2025-09-12T14:19:39.077Z	info	controller-runtime.metrics	Starting metrics server
2025-09-12T14:19:39.077Z	info	controller-runtime.metrics	Serving metrics server	{"bindAddress": "0.0.0.0:9090", "secure": false}
2025-09-12T14:19:39.077Z	info	starting server	{"name": "pprof", "addr": "[::]:8888"}
2025-09-12T14:19:39.077Z	info	starting server	{"name": "health probe", "addr": "[::]:8081"}
I0912 14:19:39.179218       1 leaderelection.go:257] attempting to acquire leader lease test-monitoring/f75f3bba.integreatly.org...
I0912 14:19:55.211363       1 leaderelection.go:271] successfully acquired lease test-monitoring/f75f3bba.integreatly.org
2025-09-12T14:19:55.212Z	info	Starting EventSource	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "source": "kind source: *v1.ConfigMap"}
2025-09-12T14:19:55.213Z	info	Starting EventSource	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "source": "kind source: *v1beta1.GrafanaDatasource"}
2025-09-12T14:19:55.213Z	info	Starting EventSource	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "source": "kind source: *v1beta1.Grafana"}
2025-09-12T14:19:55.213Z	info	Starting EventSource	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "source": "kind source: *v1.Deployment"}
2025-09-12T14:19:55.214Z	info	Starting EventSource	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "source": "kind source: *v1.ConfigMap"}
2025-09-12T14:19:55.215Z	info	Starting EventSource	{"controller": "grafanalibrarypanel", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaLibraryPanel", "source": "kind source: *v1.ConfigMap"}
2025-09-12T14:19:55.216Z	info	Starting EventSource	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "source": "kind source: *v1beta1.GrafanaDashboard"}
2025-09-12T14:19:55.216Z	info	Starting EventSource	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "source": "kind source: *v1beta1.GrafanaFolder"}
2025-09-12T14:19:55.217Z	info	Starting EventSource	{"controller": "grafanacontactpoint", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaContactPoint", "source": "kind source: *v1beta1.GrafanaContactPoint"}
2025-09-12T14:19:55.217Z	info	Starting EventSource	{"controller": "grafanalibrarypanel", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaLibraryPanel", "source": "kind source: *v1beta1.GrafanaLibraryPanel"}
2025-09-12T14:19:55.217Z	info	Starting EventSource	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup", "source": "kind source: *v1beta1.GrafanaAlertRuleGroup"}
2025-09-12T14:19:55.218Z	info	Starting EventSource	{"controller": "grafananotificationtemplate", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationTemplate", "source": "kind source: *v1beta1.GrafanaNotificationTemplate"}
2025-09-12T14:19:55.219Z	info	Starting EventSource	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy", "source": "kind source: *v1beta1.GrafanaNotificationPolicyRoute"}
2025-09-12T14:19:55.219Z	info	Starting EventSource	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy", "source": "kind source: *v1beta1.GrafanaNotificationPolicy"}
2025-09-12T14:19:55.219Z	info	Starting EventSource	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy", "source": "kind source: *v1beta1.GrafanaContactPoint"}
2025-09-12T14:19:55.220Z	info	Starting EventSource	{"controller": "grafanamutetiming", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaMuteTiming", "source": "kind source: *v1beta1.GrafanaMuteTiming"}
2025-09-12T14:19:55.314Z	info	Starting Controller	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource"}
2025-09-12T14:19:55.314Z	info	Starting workers	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "worker count": 1}
2025-09-12T14:19:55.315Z	info	Starting Controller	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana"}
2025-09-12T14:19:55.315Z	info	Starting workers	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "worker count": 1}
2025-09-12T14:19:55.317Z	info	Starting Controller	{"controller": "grafanalibrarypanel", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaLibraryPanel"}
2025-09-12T14:19:55.317Z	info	Starting workers	{"controller": "grafanalibrarypanel", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaLibraryPanel", "worker count": 1}
2025-09-12T14:19:55.317Z	info	Starting Controller	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard"}
2025-09-12T14:19:55.317Z	info	Starting workers	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "worker count": 1}
2025-09-12T14:19:55.318Z	info	Starting Controller	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder"}
2025-09-12T14:19:55.318Z	info	Starting workers	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder", "worker count": 1}
2025-09-12T14:19:55.318Z	info	Starting Controller	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup"}
2025-09-12T14:19:55.318Z	info	Starting workers	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup", "worker count": 1}
2025-09-12T14:19:55.319Z	info	Starting Controller	{"controller": "grafananotificationtemplate", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationTemplate"}
2025-09-12T14:19:55.319Z	info	Starting workers	{"controller": "grafananotificationtemplate", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationTemplate", "worker count": 1}
2025-09-12T14:19:55.319Z	info	Starting Controller	{"controller": "grafanacontactpoint", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaContactPoint"}
2025-09-12T14:19:55.319Z	info	Starting workers	{"controller": "grafanacontactpoint", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaContactPoint", "worker count": 1}
2025-09-12T14:19:55.320Z	info	Starting Controller	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy"}
2025-09-12T14:19:55.320Z	info	Starting workers	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy", "worker count": 1}
2025-09-12T14:19:55.321Z	info	Starting Controller	{"controller": "grafanamutetiming", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaMuteTiming"}
2025-09-12T14:19:55.321Z	info	Starting workers	{"controller": "grafanamutetiming", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaMuteTiming", "worker count": 1}
2025-09-12T14:20:05.221Z	info	GrafanaReconciler	Grafana status sync complete
2025-09-12T14:20:36.338Z	info	Stopping and waiting for non leader election runnables
2025-09-12T14:20:36.338Z	info	Stopping and waiting for leader election runnables
2025-09-12T14:20:36.338Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafanamutetiming", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaMuteTiming"}
2025-09-12T14:20:36.338Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy"}
2025-09-12T14:20:36.338Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafanacontactpoint", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaContactPoint"}
2025-09-12T14:20:36.338Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafananotificationtemplate", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationTemplate"}
2025-09-12T14:20:36.338Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup"}
2025-09-12T14:20:36.338Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder"}
2025-09-12T14:20:36.339Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard"}
2025-09-12T14:20:36.339Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafanalibrarypanel", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaLibraryPanel"}
2025-09-12T14:20:36.339Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana"}
2025-09-12T14:20:36.339Z	info	Shutdown signal received, waiting for all workers to finish	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafanamutetiming", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaMuteTiming"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafananotificationtemplate", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationTemplate"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafanaalertrulegroup", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaAlertRuleGroup"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafanacontactpoint", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaContactPoint"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafanafolder", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaFolder"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafanalibrarypanel", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaLibraryPanel"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource"}
2025-09-12T14:20:36.339Z	info	All workers finished	{"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana"}
2025-09-12T14:20:36.339Z	info	Stopping and waiting for caches
2025-09-12T14:20:36.339Z	info	Stopping and waiting for webhooks
2025-09-12T14:20:36.339Z	info	Stopping and waiting for HTTP servers
2025-09-12T14:20:36.340Z	info	shutting down server	{"name": "health probe", "addr": "[::]:8081"}
2025-09-12T14:20:36.340Z	info	shutting down server	{"name": "pprof", "addr": "[::]:8888"}
2025-09-12T14:20:36.340Z	info	controller-runtime.metrics	Shutting down metrics server with timeout of 1 minute
2025-09-12T14:20:36.340Z	info	Wait completed, proceeding to shutdown the manager
2025-09-12T14:20:36.340Z	info	setup	SIGTERM request gotten, shutting down operator	{"version": "v5.19.4"}

Screenshots
We can attach any required screenshots upon request.

Runtime (please complete the following information):

• OS: Linux/Fedora CoreOS 38.20230902.3.0
• Grafana Operator Version: grafana-operator v5.19.4
• Environment: OpenShift (specifically OKD with community operators) version 4.13.0-0.okd-2023-09-30-084937
• Deployment type: Openshift OLM
• Other: OpenShift installed as UPI (static-ip virtualized machines)

Additional context

This bug was already reported here #1294 , but it is not solved yet and the existing solutions didn't fix the problem.

The image in use for the CrashLoopBackOff pod is "ghcr.io/grafana/grafana-operator@sha256:b678bc3faad11f4ef53c240e2a650ce939b9e8920dd72a47681b2efcadc7b837" and is successfully pulled:

Successfully pulled image "ghcr.io/grafana/grafana-operator@sha256:b678bc3faad11f4ef53c240e2a650ce939b9e8920dd72a47681b2efcadc7b837" in 935.02102ms (935.044798ms including waiting)

Here is also the YAML for the failing deployment:

kind: Deployment
apiVersion: apps/v1
metadata:
  annotations:
    deployment.kubernetes.io/revision: '1'
  resourceVersion: '392738075'
  name: grafana-operator-controller-manager-v5
  uid: e48d07b8-327a-4242-ad25-c5679e660dbf
  creationTimestamp: '2025-09-15T07:39:01Z'
  generation: 2
  namespace: openshift-operators
  ownerReferences:
    - apiVersion: operators.coreos.com/v1alpha1
      kind: ClusterServiceVersion
      name: grafana-operator.v5.19.4
      uid: e5a3359d-8211-4b09-b1b9-414adf5e9961
      controller: false
      blockOwnerDeletion: false
  labels:
    app.kubernetes.io/managed-by: olm
    app.kubernetes.io/name: grafana-operator
    olm.deployment-spec-hash: 76c4999bf5
    olm.owner: grafana-operator.v5.19.4
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: openshift-operators
    operators.coreos.com/grafana-operator.openshift-operators: ''
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: olm
      app.kubernetes.io/name: grafana-operator
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/managed-by: olm
        app.kubernetes.io/name: grafana-operator
      annotations:
        operators.operatorframework.io/builder: operator-sdk-v1.32.0
        operators.operatorframework.io/project_layout: go.kubebuilder.io/v3
        olm.targetNamespaces: ''
        operatorframework.io/properties: >-
          {"properties":[{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"Grafana","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaAlertRuleGroup","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaContactPoint","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaDashboard","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaDatasource","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaFolder","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaLibraryPanel","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaMuteTiming","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaNotificationPolicy","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaNotificationPolicyRoute","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaNotificationTemplate","version":"v1beta1"}},{"type":"olm.package","value":{"packageName":"grafana-operator","version":"5.19.4"}}]}
        repository: 'https://github.com/grafana/grafana-operator'
        support: Grafana Labs
        alm-examples: |-
          [
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "Grafana",
              "metadata": {
                "labels": {
                  "dashboards": "grafana-a",
                  "folders": "grafana-a"
                },
                "name": "grafana-a"
              },
              "spec": {
                "config": {
                  "auth": {
                    "disable_login_form": "false"
                  },
                  "log": {
                    "mode": "console"
                  },
                  "security": {
                    "admin_password": "start",
                    "admin_user": "root"
                  }
                }
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaAlertRuleGroup",
              "metadata": {
                "name": "grafanaalertrulegroup-sample"
              },
              "spec": {
                "folderRef": "test-folder-from-operator",
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "interval": "5m",
                "rules": [
                  {
                    "condition": "B",
                    "data": [
                      {
                        "datasourceUid": "grafanacloud-demoinfra-prom",
                        "model": {
                          "datasource": {
                            "type": "prometheus",
                            "uid": "grafanacloud-demoinfra-prom"
                          },
                          "editorMode": "code",
                          "expr": "weather_temp_c{}",
                          "instant": true,
                          "intervalMs": 1000,
                          "legendFormat": "__auto",
                          "maxDataPoints": 43200,
                          "range": false,
                          "refId": "A"
                        },
                        "refId": "A",
                        "relativeTimeRange": {
                          "from": 600
                        }
                      },
                      {
                        "datasourceUid": "__expr__",
                        "model": {
                          "conditions": [
                            {
                              "evaluator": {
                                "params": [
                                  0
                                ],
                                "type": "gt"
                              },
                              "operator": {
                                "type": "and"
                              },
                              "query": {
                                "params": [
                                  "C"
                                ]
                              },
                              "reducer": {
                                "params": [],
                                "type": "last"
                              },
                              "type": "query"
                            }
                          ],
                          "datasource": {
                            "type": "__expr__",
                            "uid": "__expr__"
                          },
                          "expression": "A",
                          "intervalMs": 1000,
                          "maxDataPoints": 43200,
                          "refId": "B",
                          "type": "threshold"
                        },
                        "refId": "B",
                        "relativeTimeRange": {
                          "from": 600
                        }
                      }
                    ],
                    "execErrState": "Error",
                    "for": "5m0s",
                    "noDataState": "NoData",
                    "title": "Temperature below freezing",
                    "uid": "4843de5c-4f8a-4af0-9509-23526a04faf8"
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaContactPoint",
              "metadata": {
                "labels": {
                  "app.kubernetes.io/created-by": "grafana-operator",
                  "app.kubernetes.io/instance": "grafanacontactpoint-sample",
                  "app.kubernetes.io/managed-by": "kustomize",
                  "app.kubernetes.io/name": "grafanacontactpoint",
                  "app.kubernetes.io/part-of": "grafana-operator"
                },
                "name": "grafanacontactpoint-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "name": "grafanacontactpoint-sample",
                "settings": {
                  "email": null
                },
                "type": "email"
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaDashboard",
              "metadata": {
                "name": "grafanadashboard-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "json": "{\n\n  \"id\": null,\n  \"title\": \"Simple Dashboard\",\n  \"tags\": [],\n  \"style\": \"dark\",\n  \"timezone\": \"browser\",\n  \"editable\": true,\n  \"hideControls\": false,\n  \"graphTooltip\": 1,\n  \"panels\": [],\n  \"time\": {\n    \"from\": \"now-6h\",\n    \"to\": \"now\"\n  },\n  \"timepicker\": {\n    \"time_options\": [],\n    \"refresh_intervals\": []\n  },\n  \"templating\": {\n    \"list\": []\n  },\n  \"annotations\": {\n    \"list\": []\n  },\n  \"refresh\": \"5s\",\n  \"schemaVersion\": 17,\n  \"version\": 0,\n  \"links\": []\n}\n"
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaDatasource",
              "metadata": {
                "name": "grafanadatasource-sample"
              },
              "spec": {
                "datasource": {
                  "access": "proxy",
                  "isDefault": true,
                  "jsonData": {
                    "timeInterval": "5s",
                    "tlsSkipVerify": true
                  },
                  "name": "prometheus",
                  "type": "prometheus",
                  "url": "http://prometheus-service:9090"
                },
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "plugins": [
                  {
                    "name": "grafana-clock-panel",
                    "version": "1.3.0"
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaFolder",
              "metadata": {
                "name": "grafanafolder-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "title": "Example Folder"
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaLibraryPanel",
              "metadata": {
                "name": "grafana-library-panel-inline-envs"
              },
              "spec": {
                "envs": [
                  {
                    "name": "CUSTOM_RANGE_ENV",
                    "value": "now - 12h"
                  }
                ],
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "jsonnet": "local myRange = std.extVar('CUSTOM_RANGE_ENV'); {\n\n  model: {}\n}\n",
                "plugins": [
                  {
                    "name": "grafana-piechart-panel",
                    "version": "1.3.9"
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaMuteTiming",
              "metadata": {
                "name": "mutetiming-sample"
              },
              "spec": {
                "editable": false,
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "name": "mutetiming-sample",
                "time_intervals": [
                  {
                    "days_of_month": [
                      "1",
                      "15"
                    ],
                    "location": "Asia/Shanghai",
                    "times": [
                      {
                        "end_time": "06:00",
                        "start_time": "00:00"
                      }
                    ],
                    "weekdays": [
                      "saturday"
                    ]
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaNotificationPolicy",
              "metadata": {
                "name": "grafananotificationpolicy-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "route": {
                  "group_by": [
                    "grafana_folder",
                    "alertname"
                  ],
                  "receiver": "Grafana Cloud OnCall",
                  "routes": [
                    {
                      "object_matchers": [
                        [
                          "foo",
                          "=",
                          "bar"
                        ]
                      ],
                      "receiver": "grafana-default-email",
                      "routes": [
                        {
                          "object_matchers": [
                            [
                              "severity",
                              "=",
                              "critical"
                            ]
                          ],
                          "receiver": "Grafana Cloud OnCall"
                        }
                      ]
                    }
                  ]
                }
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaNotificationPolicyRoute",
              "metadata": {
                "labels": {
                  "app.kubernetes.io/created-by": "grafana-operator",
                  "app.kubernetes.io/instance": "grafananotificationpolicyroute-sample",
                  "app.kubernetes.io/managed-by": "kustomize",
                  "app.kubernetes.io/name": "grafananotificationpolicyroute",
                  "app.kubernetes.io/part-of": "grafana-operator"
                },
                "name": "grafananotificationpolicyroute-sample"
              },
              "spec": null
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaNotificationTemplate",
              "metadata": {
                "name": "test"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "name": "test",
                "template": "{{ define \"SlackAlert\" }}\n  [{{.Status}}] {{ .Labels.alertname }}\n  {{ .Annotations.AlertValues }}\n{{ end }}\n\n{{ define \"SlackAlertMessage\" }}\n  {{ if gt (len .Alerts.Firing) 0 }}\n    {{ len .Alerts.Firing }} firing:\n    {{ range .Alerts.Firing }} {{ template \"SlackAlert\" . }} {{ end }}\n  {{ end }}\n  {{ if gt (len .Alerts.Resolved) 0 }}\n    {{ len .Alerts.Resolved }} resolved:\n    {{ range .Alerts.Resolved }} {{ template \"SlackAlert\" . }} {{ end }}\n  {{ end }}\n{{ end }}\n\n{{ template \"SlackAlertMessage\" . }}\n"
              }
            }
          ]
        capabilities: Basic Install
        olm.operatorNamespace: openshift-operators
        containerImage: >-
          ghcr.io/grafana/grafana-operator@sha256:b678bc3faad11f4ef53c240e2a650ce939b9e8920dd72a47681b2efcadc7b837
        createdAt: '2025-08-21T08:38:00Z'
        categories: Monitoring
        description: 'Deploys and manages Grafana instances, dashboards and data sources'
        olm.operatorGroup: global-operators
    spec:
      containers:
        - resources:
            limits:
              cpu: 200m
              memory: 550Mi
            requests:
              cpu: 100m
              memory: 20Mi
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8081
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          name: manager
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8081
              scheme: HTTP
            initialDelaySeconds: 15
            timeoutSeconds: 1
            periodSeconds: 20
            successThreshold: 1
            failureThreshold: 3
          env:
            - name: RELATED_IMAGE_GRAFANA
              value: >-
                docker.io/grafana/grafana@sha256:6ac590e7cabc2fbe8d7b8fc1ce9c9f0582177b334e0df9c927ebd9670469440f
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: 'metadata.annotations[''olm.targetNamespaces'']'
            - name: OPERATOR_CONDITION_NAME
              value: grafana-operator.v5.19.4
          securityContext:
            allowPrivilegeEscalation: false
          ports:
            - name: metrics
              containerPort: 9090
              protocol: TCP
            - name: pprof
              containerPort: 8888
              protocol: TCP
          imagePullPolicy: Always
          terminationMessagePolicy: File
          image: >-
            ghcr.io/grafana/grafana-operator@sha256:b678bc3faad11f4ef53c240e2a650ce939b9e8920dd72a47681b2efcadc7b837
          args:
            - '--health-probe-bind-address=:8081'
            - '--metrics-bind-address=0.0.0.0:9090'
            - '--leader-elect'
      restartPolicy: Always
      terminationGracePeriodSeconds: 10
      dnsPolicy: ClusterFirst
      serviceAccountName: grafana-operator-controller-manager
      serviceAccount: grafana-operator-controller-manager
      securityContext:
        runAsNonRoot: true
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 1
  progressDeadlineSeconds: 600
status:
  observedGeneration: 2
  replicas: 1
  updatedReplicas: 1
  unavailableReplicas: 1
  conditions:
    - type: Available
      status: 'False'
      lastUpdateTime: '2025-09-15T07:39:01Z'
      lastTransitionTime: '2025-09-15T07:39:01Z'
      reason: MinimumReplicasUnavailable
      message: Deployment does not have minimum availability.
    - type: Progressing
      status: 'True'
      lastUpdateTime: '2025-09-15T07:39:01Z'
      lastTransitionTime: '2025-09-15T07:39:01Z'
      reason: ReplicaSetUpdated
      message: >-
        ReplicaSet "grafana-operator-controller-manager-v5-5595b667f6" is
        progressing.

Already tried removing/reinstalling the operator a few times, unfortunately it always gets stuck at this stage.

Any help would be greatly appreciated in solving this issue, so far we could't find any successfull workarounds. Thank you!

[theSuess]

Can you try to manually perform a request to the :8081/healthz endpoint (using the IP, not the service name)? I'd suspect network policies to be the issue here

[glowing-axolotl]

Sure, let me try right now

[glowing-axolotl]

List of pods in the namespace with IPs:

# oc get pods -n openshift-operators -o wide
NAME                                                      READY   STATUS    RESTARTS      AGE     IP             NODE        NOMINATED NODE   READINESS GATES
grafana-operator-controller-manager-v5-5595b667f6-wdgkg   0/1     Running   5 (13s ago)   5m53s   10.247.7.119   worker-06   <none>           <none>
test-debug-rqbqm                                          1/1     Running   0             7m23s   10.247.7.118   worker-06   <none>           <none>

Ping working:

curl not working:

List of NetworkPolicies (there are none configured, we are using kubeadmin):

# oc get networkpolicies  -A
No resources found

We're also trying to clone the deployment and remove/increase the probes timeout, will post updates.

[glowing-axolotl]

Here is the full YAML of the cloned deployment for testing (mostly only changed the probes to have a failureThreshold of 999):

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  name: grafana-operator-controller-manager-v5-deb
  namespace: openshift-operators
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: grafana-operator-deb
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        alm-examples: |-
          [
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "Grafana",
              "metadata": {
                "labels": {
                  "dashboards": "grafana-a",
                  "folders": "grafana-a"
                },
                "name": "grafana-a"
              },
              "spec": {
                "config": {
                  "auth": {
                    "disable_login_form": "false"
                  },
                  "log": {
                    "mode": "console"
                  },
                  "security": {
                    "admin_password": "start",
                    "admin_user": "root"
                  }
                }
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaAlertRuleGroup",
              "metadata": {
                "name": "grafanaalertrulegroup-sample"
              },
              "spec": {
                "folderRef": "test-folder-from-operator",
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "interval": "5m",
                "rules": [
                  {
                    "condition": "B",
                    "data": [
                      {
                        "datasourceUid": "grafanacloud-demoinfra-prom",
                        "model": {
                          "datasource": {
                            "type": "prometheus",
                            "uid": "grafanacloud-demoinfra-prom"
                          },
                          "editorMode": "code",
                          "expr": "weather_temp_c{}",
                          "instant": true,
                          "intervalMs": 1000,
                          "legendFormat": "__auto",
                          "maxDataPoints": 43200,
                          "range": false,
                          "refId": "A"
                        },
                        "refId": "A",
                        "relativeTimeRange": {
                          "from": 600
                        }
                      },
                      {
                        "datasourceUid": "__expr__",
                        "model": {
                          "conditions": [
                            {
                              "evaluator": {
                                "params": [
                                  0
                                ],
                                "type": "gt"
                              },
                              "operator": {
                                "type": "and"
                              },
                              "query": {
                                "params": [
                                  "C"
                                ]
                              },
                              "reducer": {
                                "params": [],
                                "type": "last"
                              },
                              "type": "query"
                            }
                          ],
                          "datasource": {
                            "type": "__expr__",
                            "uid": "__expr__"
                          },
                          "expression": "A",
                          "intervalMs": 1000,
                          "maxDataPoints": 43200,
                          "refId": "B",
                          "type": "threshold"
                        },
                        "refId": "B",
                        "relativeTimeRange": {
                          "from": 600
                        }
                      }
                    ],
                    "execErrState": "Error",
                    "for": "5m0s",
                    "noDataState": "NoData",
                    "title": "Temperature below freezing",
                    "uid": "4843de5c-4f8a-4af0-9509-23526a04faf8"
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaContactPoint",
              "metadata": {
                "labels": {
                  "app.kubernetes.io/created-by": "grafana-operator",
                  "app.kubernetes.io/instance": "grafanacontactpoint-sample",
                  "app.kubernetes.io/managed-by": "kustomize",
                  "app.kubernetes.io/name": "grafanacontactpoint",
                  "app.kubernetes.io/part-of": "grafana-operator"
                },
                "name": "grafanacontactpoint-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "name": "grafanacontactpoint-sample",
                "settings": {
                  "email": null
                },
                "type": "email"
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaDashboard",
              "metadata": {
                "name": "grafanadashboard-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "json": "{\n\n  \"id\": null,\n  \"title\": \"Simple Dashboard\",\n  \"tags\": [],\n  \"style\": \"dark\",\n  \"timezone\": \"browser\",\n  \"editable\": true,\n  \"hideControls\": false,\n  \"graphTooltip\": 1,\n  \"panels\": [],\n  \"time\": {\n    \"from\": \"now-6h\",\n    \"to\": \"now\"\n  },\n  \"timepicker\": {\n    \"time_options\": [],\n    \"refresh_intervals\": []\n  },\n  \"templating\": {\n    \"list\": []\n  },\n  \"annotations\": {\n    \"list\": []\n  },\n  \"refresh\": \"5s\",\n  \"schemaVersion\": 17,\n  \"version\": 0,\n  \"links\": []\n}\n"
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaDatasource",
              "metadata": {
                "name": "grafanadatasource-sample"
              },
              "spec": {
                "datasource": {
                  "access": "proxy",
                  "isDefault": true,
                  "jsonData": {
                    "timeInterval": "5s",
                    "tlsSkipVerify": true
                  },
                  "name": "prometheus",
                  "type": "prometheus",
                  "url": "http://prometheus-service:9090"
                },
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "plugins": [
                  {
                    "name": "grafana-clock-panel",
                    "version": "1.3.0"
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaFolder",
              "metadata": {
                "name": "grafanafolder-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana-a"
                  }
                },
                "title": "Example Folder"
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaLibraryPanel",
              "metadata": {
                "name": "grafana-library-panel-inline-envs"
              },
              "spec": {
                "envs": [
                  {
                    "name": "CUSTOM_RANGE_ENV",
                    "value": "now - 12h"
                  }
                ],
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "jsonnet": "local myRange = std.extVar('CUSTOM_RANGE_ENV'); {\n\n  model: {}\n}\n",
                "plugins": [
                  {
                    "name": "grafana-piechart-panel",
                    "version": "1.3.9"
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaMuteTiming",
              "metadata": {
                "name": "mutetiming-sample"
              },
              "spec": {
                "editable": false,
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "name": "mutetiming-sample",
                "time_intervals": [
                  {
                    "days_of_month": [
                      "1",
                      "15"
                    ],
                    "location": "Asia/Shanghai",
                    "times": [
                      {
                        "end_time": "06:00",
                        "start_time": "00:00"
                      }
                    ],
                    "weekdays": [
                      "saturday"
                    ]
                  }
                ]
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaNotificationPolicy",
              "metadata": {
                "name": "grafananotificationpolicy-sample"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "route": {
                  "group_by": [
                    "grafana_folder",
                    "alertname"
                  ],
                  "receiver": "Grafana Cloud OnCall",
                  "routes": [
                    {
                      "object_matchers": [
                        [
                          "foo",
                          "=",
                          "bar"
                        ]
                      ],
                      "receiver": "grafana-default-email",
                      "routes": [
                        {
                          "object_matchers": [
                            [
                              "severity",
                              "=",
                              "critical"
                            ]
                          ],
                          "receiver": "Grafana Cloud OnCall"
                        }
                      ]
                    }
                  ]
                }
              }
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaNotificationPolicyRoute",
              "metadata": {
                "labels": {
                  "app.kubernetes.io/created-by": "grafana-operator",
                  "app.kubernetes.io/instance": "grafananotificationpolicyroute-sample",
                  "app.kubernetes.io/managed-by": "kustomize",
                  "app.kubernetes.io/name": "grafananotificationpolicyroute",
                  "app.kubernetes.io/part-of": "grafana-operator"
                },
                "name": "grafananotificationpolicyroute-sample"
              },
              "spec": null
            },
            {
              "apiVersion": "grafana.integreatly.org/v1beta1",
              "kind": "GrafanaNotificationTemplate",
              "metadata": {
                "name": "test"
              },
              "spec": {
                "instanceSelector": {
                  "matchLabels": {
                    "dashboards": "grafana"
                  }
                },
                "name": "test",
                "template": "{{ define \"SlackAlert\" }}\n  [{{.Status}}] {{ .Labels.alertname }}\n  {{ .Annotations.AlertValues }}\n{{ end }}\n\n{{ define \"SlackAlertMessage\" }}\n  {{ if gt (len .Alerts.Firing) 0 }}\n    {{ len .Alerts.Firing }} firing:\n    {{ range .Alerts.Firing }} {{ template \"SlackAlert\" . }} {{ end }}\n  {{ end }}\n  {{ if gt (len .Alerts.Resolved) 0 }}\n    {{ len .Alerts.Resolved }} resolved:\n    {{ range .Alerts.Resolved }} {{ template \"SlackAlert\" . }} {{ end }}\n  {{ end }}\n{{ end }}\n\n{{ template \"SlackAlertMessage\" . }}\n"
              }
            }
          ]
        capabilities: Basic Install
        categories: Monitoring
        containerImage: ghcr.io/grafana/grafana-operator@sha256:b678bc3faad11f4ef53c240e2a650ce939b9e8920dd72a47681b2efcadc7b837
        createdAt: "2025-08-21T08:38:00Z"
        description: Deploys and manages Grafana instances, dashboards and data sources
        olm.operatorGroup: global-operators
        olm.operatorNamespace: openshift-operators
        olm.targetNamespaces: ""
        operatorframework.io/properties: '{"properties":[{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"Grafana","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaAlertRuleGroup","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaContactPoint","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaDashboard","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaDatasource","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaFolder","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaLibraryPanel","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaMuteTiming","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaNotificationPolicy","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaNotificationPolicyRoute","version":"v1beta1"}},{"type":"olm.gvk","value":{"group":"grafana.integreatly.org","kind":"GrafanaNotificationTemplate","version":"v1beta1"}},{"type":"olm.package","value":{"packageName":"grafana-operator","version":"5.19.4"}}]}'
        operators.operatorframework.io/builder: operator-sdk-v1.32.0
        operators.operatorframework.io/project_layout: go.kubebuilder.io/v3
        repository: https://github.com/grafana/grafana-operator
        support: Grafana Labs
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: grafana-operator-deb
    spec:
      containers:
      - args:
        - --health-probe-bind-address=:8081
        - --metrics-bind-address=0.0.0.0:9090
        - --leader-elect
        env:
        - name: RELATED_IMAGE_GRAFANA
          value: docker.io/grafana/grafana@sha256:6ac590e7cabc2fbe8d7b8fc1ce9c9f0582177b334e0df9c927ebd9670469440f
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.annotations['olm.targetNamespaces']
        - name: OPERATOR_CONDITION_NAME
          value: grafana-operator.v5.19.4
        image: ghcr.io/grafana/grafana-operator@sha256:b678bc3faad11f4ef53c240e2a650ce939b9e8920dd72a47681b2efcadc7b837
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 999
          httpGet:
            path: /healthz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 1
        name: manager
        ports:
        - containerPort: 9090
          name: metrics
          protocol: TCP
        - containerPort: 8888
          name: pprof
          protocol: TCP
        readinessProbe:
          failureThreshold: 999
          httpGet:
            path: /readyz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 200m
            memory: 550Mi
          requests:
            cpu: 100m
            memory: 20Mi
        securityContext:
          allowPrivilegeEscalation: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
      serviceAccount: grafana-operator-controller-manager
      serviceAccountName: grafana-operator-controller-manager
      terminationGracePeriodSeconds: 10

Here is the YAML for the network troubleshooting pod:

``` apiVersion: batch/v1 kind: Job metadata: name: test-debug spec: ttlSecondsAfterFinished: 1209600 # 14 days template: spec: containers: - name: test-container image: wbitt/network-multitool:openshift imagePullPolicy: IfNotPresent command: - /bin/sh - -c - sleep 40m restartPolicy: Never ```

Here is the list of pods:

# oc get pods -n openshift-operators -o wide
NAME                                                          READY   STATUS             RESTARTS         AGE   IP             NODE        NOMINATED NODE   READINESS GATES
grafana-operator-controller-manager-v5-5595b667f6-wdgkg       0/1     CrashLoopBackOff   13 (3m27s ago)   36m   10.247.7.119   worker-06   <none>           <none>
grafana-operator-controller-manager-v5-deb-5984f4f885-z6vfj   0/1     Running            0                12m   10.247.7.123   worker-06   <none>           <none>
test-debug-rqbqm                                              1/1     Running            0                37m   10.247.7.118   worker-06   <none>           <none>

Here are the two curls from the network troubleshooting pod:

```bash ~ $ ping 10.247.7.123 PING 10.247.7.123 (10.247.7.123) 56(84) bytes of data. 64 bytes from 10.247.7.123: icmp_seq=1 ttl=64 time=3.11 ms 64 bytes from 10.247.7.123: icmp_seq=2 ttl=64 time=1.22 ms ^C --- 10.247.7.123 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 1.218/2.162/3.106/0.944 ms ~ $ curl -v -k http://10.247.7.123:8081/readyz * processing: http://10.247.7.123:8081/readyz * Trying 10.247.7.123:8081... * Connected to 10.247.7.123 (10.247.7.123) port 8081 > GET /readyz HTTP/1.1 > Host: 10.247.7.123:8081 > User-Agent: curl/8.2.1 > Accept: */* > * Recv failure: Connection reset by peer * Closing connection curl: (56) Recv failure: Connection reset by peer

~ $ curl -v -k http://10.247.7.123:8081/healthz

• processing: http://10.247.7.123:8081/healthz
• Trying 10.247.7.123:8081...
• Connected to 10.247.7.123 (10.247.7.123) port 8081

GET /healthz HTTP/1.1
Host: 10.247.7.123:8081
User-Agent: curl/8.2.1
Accept: /

• Recv failure: Connection reset by peer
• Closing connection
  curl: (56) Recv failure: Connection reset by peer

</details>

[glowing-axolotl]

Hello @theSuess ,

Could I ask you if you have other troubleshooting tips?

If needed, I can setup a separate debugging environment/pod. If I understood the problem correctly, the problem should be somewhere around https://github.com/grafana/grafana-operator/blob/master/main.go between the "// Determine Operator scope" (since there is a log for this section) and the call to mgr.AddHealthzCheck . Would this guess be correct?

Thank you

[Baarsgaard]

@glowing-axolotl Would you be able to delete the pod in CrashLoopBackoff
This should trigger a reschedule.
Then when it re-enters the crash loop gather the following:
kubectl describe pod -n openshift-operators grafana-operator-controller-manager-v5-xxxxxxxx-xxxxxx

If you have no Grafanas in the cluster, there should be no reconciliation logs from the operator, which is expected.
But it does look like the default healthcheck cannot reach the operator and is killing it when it's done starting up.
kubectl describe should list the events at the bottom which confirms/denies that.
If not, kubectl events -n openshift-operators | grep grafana could find more.

If it turns out to be the healthcheck, does editing/patching the deployment and removing the healthchecks allow the operator to stay running?
If so, I would investigate why OpenShift cannot reach the pod on the given port.

[stefanocogoi]

Good morning,
I am replying on behalf of @glowing-axolotl as we work on the same openshift cluster. Below I attach a file with the output of the describe command and a file with the output of the get events command.

grafana-events.txt
describe-grafana-operator-controller.txt

thx

[glowing-axolotl]

@glowing-axolotl If it turns out to be the healthcheck, does editing/patching the deployment and removing the healthchecks allow the operator to stay running? If so, I would investigate why OpenShift cannot reach the pod on the given port.

Hi @Baarsgaard ,

In the following message we also tried the following:

Here is the full YAML of the cloned deployment for testing (mostly only changed the probes to have a failureThreshold of 999):
...

# oc get pods -n openshift-operators -o wide
NAME                                                          READY   STATUS             RESTARTS         AGE   IP             NODE        NOMINATED NODE   READINESS GATES
grafana-operator-controller-manager-v5-5595b667f6-wdgkg       0/1     CrashLoopBackOff   13 (3m27s ago)   36m   10.247.7.119   worker-06   <none>           <none>
grafana-operator-controller-manager-v5-deb-5984f4f885-z6vfj   0/1     Running            0                12m   10.247.7.123   worker-06   <none>           <none>
test-debug-rqbqm                                              1/1     Running            0                37m   10.247.7.118   worker-06   <none>           <none>

Here are the two curls from the network troubleshooting pod:

~ $ ping 10.247.7.123
PING 10.247.7.123 (10.247.7.123) 56(84) bytes of data.
64 bytes from 10.247.7.123: icmp_seq=1 ttl=64 time=3.11 ms
64 bytes from 10.247.7.123: icmp_seq=2 ttl=64 time=1.22 ms
^C
--- 10.247.7.123 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 1.218/2.162/3.106/0.944 ms
~ $ curl -v -k http://10.247.7.123:8081/readyz
* processing: http://10.247.7.123:8081/readyz
*   Trying 10.247.7.123:8081...
* Connected to 10.247.7.123 (10.247.7.123) port 8081
> GET /readyz HTTP/1.1
> Host: 10.247.7.123:8081
> User-Agent: curl/8.2.1
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

~ $ curl -v -k http://10.247.7.123:8081/healthz
* processing: http://10.247.7.123:8081/healthz
*   Trying 10.247.7.123:8081...
* Connected to 10.247.7.123 (10.247.7.123) port 8081
> GET /healthz HTTP/1.1
> Host: 10.247.7.123:8081
> User-Agent: curl/8.2.1
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection
curl: (56) Recv failure: Connection reset by peer

But to no avail, unfortunately it seems that Go's binding of the healthcheck APIs fails somehow.

Would you have any tips on troubleshooting this issue? Is there some way to trace the code or get extremely specific debug logs for the problem? Thanks!

Otherwise we can try to setup some sort of debugging environment (once again any tips would be welcome, I was thinking of running Delve inside a pod with the source code cloned, but there might be better ways to do this).

[theSuess]

If you want to debug this further, I'd recommend you use the helm or kustomize deployment method, as this will give you the recommended setup. You can then replace the operator image with a custom image that contains the binary as well as any debugging tool you wish.

From what I can see though, this is not an issue with the operator. If you can show that this only happens with the operator (and health checks for other deployments in the same namespace work as expected), we can take a closer look

[glowing-axolotl]

If you want to debug this further, I'd recommend you use the helm or kustomize deployment method, as this will give you the recommended setup. You can then replace the operator image with a custom image that contains the binary as well as any debugging tool you wish.

Hi, just a quick update, we'll try out the helm installation and report/update this issue, possibly this Monday or a few days later.

[glowing-axolotl]

Hi @Baarsgaard @theSuess ,

Can confirm the installation also fails with helm:

# ./helm upgrade -i grafana-operator oci://ghcr.io/grafana/helm-charts/grafana-operator --version v5.19.0 -n my-clean-test-namespace
Release "grafana-operator" does not exist. Installing it now.
Pulled: ghcr.io/grafana/helm-charts/grafana-operator:v5.19.0
Digest: sha256:cb2b803815a039c3a31a529f8c5605768127c22a9131b020d09601f2e26822d2
NAME: grafana-operator
LAST DEPLOYED: Tue Sep 30 16:01:06 2025
NAMESPACE: my-clean-test-namespace
STATUS: deployed
REVISION: 1
TEST SUITE: None

oc get pods -n my-clean-test-namespace -w
NAME                                READY   STATUS    RESTARTS      AGE
grafana-operator-76677d7556-6vlp2   0/1     Running   1 (25s ago)   55s

Pod events:

oc describe pod grafana-operator-76677d7556-6vlp2 -n my-clean-test-namespace
Name:             grafana-operator-76677d7556-6vlp2
Namespace:        my-clean-test-namespace
Priority:         0
Service Account:  grafana-operator
Node:             worker-06/172.31.254.25
Start Time:       Tue, 30 Sep 2025 15:59:14 +0200
Labels:           app.kubernetes.io/component=operator
                  app.kubernetes.io/instance=grafana-operator
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=grafana-operator
                  app.kubernetes.io/part-of=grafana-operator
                  app.kubernetes.io/version=v5.19.0
                  helm.sh/chart=grafana-operator-v5.19.0
                  pod-template-hash=76677d7556
Annotations:      k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["10.247.7.33/23"],"mac_address":"0a:58:0a:f7:07:21","gateway_ips":["10.247.6.1"],"ip_address":"10.247.7.33/23"...
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "10.247.7.33"
                        ],
                        "mac": "0a:58:0a:f7:07:21",
                        "default": true,
                        "dns": {}
                    }]
                  openshift.io/scc: restricted-v2
                  seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:           Running
IP:               10.247.7.33
IPs:
  IP:           10.247.7.33
Controlled By:  ReplicaSet/grafana-operator-76677d7556
Containers:
  grafana-operator:
    Container ID:  cri-o://9487e49218bdadadc251edd85ca1df1659b07ce0b3c6787d12f03c981bb44eb5
    Image:         ghcr.io/grafana/grafana-operator:v5.19.0
    Image ID:      ghcr.io/grafana/grafana-operator@sha256:002d6bd4c10e67ef392202f25cc48e3566c96c625ab3cf930610d4a0a147b4cd
    Ports:         9090/TCP, 8888/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --health-probe-bind-address=:8081
      --metrics-bind-address=0.0.0.0:9090
      --pprof-addr=0.0.0.0:8888
      --zap-encoder=console
      --zap-log-level=info
      --zap-time-encoding=rfc3339
      --leader-elect
      --max-concurrent-reconciles=1
    State:          Running
      Started:      Tue, 30 Sep 2025 16:00:45 +0200
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 30 Sep 2025 16:00:15 +0200
      Finished:     Tue, 30 Sep 2025 16:00:44 +0200
    Ready:          False
    Restart Count:  3
    Liveness:       http-get http://:8081/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:8081/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      WATCH_NAMESPACE:
      WATCH_NAMESPACE_SELECTOR:
      WATCH_LABEL_SELECTORS:
      ENFORCE_CACHE_LABELS:      safe
      CLUSTER_DOMAIN:
    Mounts:
      /tmp/dashboards from dashboards-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x8h69 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  dashboards-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-x8h69:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Normal   Scheduled       3m42s                  default-scheduler  Successfully assigned my-clean-test-namespace/grafana-operator-76677d7556-6vlp2 to worker-06
  Normal   AddedInterface  3m42s                  multus             Add eth0 [10.247.7.33/23] from ovn-kubernetes
  Normal   Pulling         3m41s                  kubelet            Pulling image "ghcr.io/grafana/grafana-operator:v5.19.0"
  Normal   Pulled          3m35s                  kubelet            Successfully pulled image "ghcr.io/grafana/grafana-operator:v5.19.0" in 6.206900778s (6.206910737s including waiting)
  Warning  Unhealthy       3m33s                  kubelet            Readiness probe failed: Get "http://10.247.7.33:8081/readyz": read tcp 10.247.6.2:58040->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       3m32s                  kubelet            Liveness probe failed: Get "http://10.247.7.33:8081/healthz": read tcp 10.247.6.2:58070->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       3m22s                  kubelet            Liveness probe failed: Get "http://10.247.7.33:8081/healthz": read tcp 10.247.6.2:51382->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       3m22s                  kubelet            Readiness probe failed: Get "http://10.247.7.33:8081/readyz": read tcp 10.247.6.2:51388->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       3m12s                  kubelet            Readiness probe failed: Get "http://10.247.7.33:8081/readyz": read tcp 10.247.6.2:47820->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       3m12s                  kubelet            Liveness probe failed: Get "http://10.247.7.33:8081/healthz": read tcp 10.247.6.2:47808->10.247.7.33:8081: read: connection reset by peer
  Normal   Killing         3m12s                  kubelet            Container grafana-operator failed liveness probe, will be restarted
  Normal   Pulled          3m12s                  kubelet            Container image "ghcr.io/grafana/grafana-operator:v5.19.0" already present on machine
  Normal   Started         3m11s (x2 over 3m35s)  kubelet            Started container grafana-operator
  Normal   Created         3m11s (x2 over 3m35s)  kubelet            Created container grafana-operator
  Warning  Unhealthy       3m9s                   kubelet            Readiness probe failed: Get "http://10.247.7.33:8081/readyz": read tcp 10.247.6.2:47854->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       3m2s                   kubelet            Liveness probe failed: Get "http://10.247.7.33:8081/healthz": read tcp 10.247.6.2:59198->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       2m52s (x3 over 3m2s)   kubelet            (combined from similar events): Readiness probe failed: Get "http://10.247.7.33:8081/readyz": read tcp 10.247.6.2:46410->10.247.7.33:8081: read: connection reset by peer
  Warning  Unhealthy       2m42s (x6 over 3m32s)  kubelet            Readiness probe failed: Get "http://10.247.7.33:8081/readyz": EOF

Such a weirdly interesting bug. Would you be interested in taking a closer look? Let us know if that is the case. Thanks!