request failed: status: DataLoss, message: "Delivery failed", details: []

[zedgepetkepau]

Question

Hello im not able to backtrack errors that im getting. In opensearch i can confirm that im getting logs and there is connectivity. but in pods i see errors but without any explanation. Debug level does not help also. does not provide any additional information for troubleshooting. Im not sure what could be wrong. from configuration perspective everything looks fine.
These error come up on sender logs. in receiver logs i dont see any errors.

Can someone please help me to investigate the issue? or at least point where could i start looking..

Vector Config

sender:

      customConfig = {
        acknowledgements = {
          enabled = true
        }
        data_dir = "/vector-data-dir"

        # Kubernetes logs source
        sources = {
          kubernetes_logs = {
            type                           = "kubernetes_logs"
            extra_namespace_label_selector = "name!=gitlab-runners"
          }
        }

        # VRL transforms
        transforms = {
          vrl_transforms = {
            type   = "remap"
            inputs = ["kubernetes_logs"]
            source = <<-EOT
              ## Unify kubernetes app label. In source sometimes is object, sometimes string...
              del(.kubernetes.node_labels)
              if is_string(.kubernetes.pod_labels.app) {
                app = .kubernetes.pod_labels.app
                del(.kubernetes.pod_labels.app)
                .kubernetes.pod_labels.app."kubernetes.io/name" = app
              }

              ## Add additional fields required for indexing
              if !exists(.environment) {
                .environment, err = .kubernetes.pod_labels.environment || get_env_var("ENVIRONMENT") || "unknown"
              }
              .cloud.alias, err = get_env_var("GCP_PROJECT") || "unknown"
              
              ## Special handling for kube-system namespace
              if .kubernetes.pod_namespace == "kube-system" {
                .owner = "team-platform-engineering"
                .service = .kubernetes.container_name
                .product = "kube-system"
              } else {
                ## Extract service field - try multiple label sources
                .service, err = get(.kubernetes.pod_labels, ["service"]) || 
                               get(.kubernetes.pod_labels, ["app.kubernetes.io/name"]) || 
                               get(.kubernetes.pod_labels, ["app"]) || 
                               "unknown"
                
                ## Extract owner field
                .owner, err = get(.kubernetes.pod_labels, ["owner"]) || "unknown"
                
                ## Extract product field
                .product, err = get(.kubernetes.pod_labels, ["product"]) || "unknown"
              }

              ## Extract cluster name field
              .cluster_name, err = get_env_var("CLUSTER_NAME") || "unknown"

              ## Parse json message
              .log_raw = .message
              messageJson, err = is_json(.message)
              if messageJson {
                .log, err = parse_json(.message)
              }
              del(.message)

              ## add additional field to easily catch logs where timestamp is missing
              if !exists(.timestamp) {
                .notimestamp = true
              }
            EOT
          }
        }

        # Vector aggregator sink
        sinks = {
          vector_aggregator = {
            type    = "vector"
            inputs  = ["vrl_transforms"]
            address = "vector-aggregator.${lower(var.environment)}.mydomain.io.internal.:6000"
            version = "2"
            buffer = [
              {
                type       = "memory"
                max_events = 1000
                when_full  = "overflow"
              },
              {
                type      = "disk"
                max_size  = 10737418240 # 10GB
                when_full = "block"
              }
            ]
            batch = {
              max_events   = 1000
              timeout_secs = 5
            }
            request = {
              retry_attempts             = 5
              retry_initial_backoff_secs = 1
              retry_max_duration_secs    = 60
              timeout_secs               = 30
            }
          }
        }
      }

receiver:

    customConfig = {
      data_dir = "/vector-data-dir"
      acknowledgements = {
        enabled = true
      }
      api = {
        enabled = true
        address = "0.0.0.0:9000"
      }
      sources = {
        vector = {
          type    = "vector"
          address = "0.0.0.0:6000"
          version = "2"
        }
      }
      sinks = {
        console = {
          type   = "console"
          inputs = ["vector"]
          encoding = {
            codec = "json"
          }
        }
        opensearch = {
          type        = "elasticsearch"
          api_version = "v8"
          inputs      = ["vector"]
          endpoints   = ["https://opensearch-cluster-data:9200"]
          tls = {
            verify_certificate = false
            server_name        = "opensearch-cluster-data"
            verify_hostname    = true
          }
          auth = {
            strategy = "basic"
            user     = "$${ELASTICSEARCH_USERNAME}"
            password = "$${ELASTICSEARCH_PASSWORD}"
          }
          bulk = {
            index = "{{ \"{{\" }} environment {{ \"}}\" }}-{{ \"{{\" }} service {{ \"}}\" }}-vector-beat-%Y.%m.%d"
          }
          batch = {
            max_events   = 1000
            timeout_secs = 5
          }
          request = {
            retry_attempts             = 5
            retry_initial_backoff_secs = 1
            retry_max_duration_secs    = 60
            timeout_secs               = 30
          }
          buffer = [
            {
              type       = "memory"
              max_events = 5000
              when_full  = "overflow"
            },
            {
              type      = "disk"
              max_size  = 18000000000 # 18GB
              when_full = "block"
            }
          ]
        }
      }
    }

Vector Logs

2025-11-13T10:50:40.411015Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=39}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Request { source: Status { code: DataLoss, message: "Delivery failed", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 13 Nov 2025 10:50:40 GMT", "content-length": "0"} }, source: None } }) request_id=39 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2025-11-13T10:50:40.411066Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=39}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=25 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2025-11-13T10:51:03.704404Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=41}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Request failed: status: DataLoss, message: "Delivery failed", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 13 Nov 2025 10:51:03 GMT", "content-length": "0"} } internal_log_rate_limit=true
2025-11-13T10:51:03.704922Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=41}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Request { source: Status { code: DataLoss, message: "Delivery failed", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 13 Nov 2025 10:51:03 GMT", "content-length": "0"} }, source: None } }) request_id=41 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2025-11-13T10:51:03.704973Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=41}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=34 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true
2025-11-13T10:51:13.463525Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=42}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being suppressed to avoid flooding.
2025-11-13T10:51:13.463623Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=42}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] is being suppressed to avoid flooding.
2025-11-13T10:51:13.463732Z ERROR sink{component_kind="sink" component_id=vector_aggregator component_type=vector}:request{request_id=42}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.