My VRL transformation works in playground but does not work locally

[olekspickle]

Question

I'm running vector with clickhouse in docker compose.

For some reason my VRL transform works on vrl playground but not locally.
All healthchecks are passing on vector validate but no data in clickhouse, and I see normal unaltered messages being logged into console.

Vector Config

api:
  enabled: true
  address: "0.0.0.0:8686"

sources:
   # Host metrics source for system metrics (CPU, memory, disk, etc.)
  host_metrics:
    type: "host_metrics"
    collectors: ["cpu", "memory", "disk", "filesystem", "network"]

    # Transform host metrics to add tags and format for ClickHouse
transforms:
  format_host_metrics:
    inputs: ["host_metrics"]
    type: "remap"
    source: |
      .timestamp, err = to_string(del(.timestamp))
      .metric_name = .name
      .host = get_hostname() ?? "unknown"
      .source = "vector_host_metrics"

      # Extract value based on metric type
      if exists(.gauge) {
      .value = .gauge.value
      } else if exists(.counter) {
      .value = .counter.value
      } else {
      .value = 0.0
      }

      # Add metric-specific tags
      if .name == "cpu" {
      .tags = {
      "cpu": string(.tags.cpu) ?? "unknown",
      "mode": string(.tags.mode) ?? "unknown"
      }
      .value = to_float(.value) * 100 ?? 0.0  # Convert to percentage
      } else if .name == "memory" {
      .tags = {
      "state": string(.tags.state) ?? "unknown"
      }
      } else if .name == "disk" {
      .tags = {
      "device": string(.tags.device) ?? "unknown"
      }
      } else if .name == "network" {
      .tags = {
      "device": string(.tags.device) ?? "unknown"
      }
      } else if .name == "filesystem" {
      .tags = {
      "device": string(.tags.device) ?? "unknown",
      "fstype": string(.tags.fstype) ?? "unknown"
      }
      } else {
      .tags = {}
      }

      # Clean up original fields
      del(.name)
      del(.namespace)
      del(.gauge)
      del(.counter)
      del(.kind)

sinks:
  # ClickHouse sink for host metrics
  clickhouse_host_metrics:
    type: "clickhouse"
    inputs: ["format_host_metrics"]
    endpoint: "${CLICKHOUSE_ENDPOINT:-http://clickhouse:8123}"
    database: "metrics"
    table: "host"
    auth:
      strategy: "basic"
      user: "${CLICKHOUSE_USER}"
      password: "${CLICKHOUSE_PASSWORD}"
      skip_unknown_fields: true

  # Console sink for debugging
  console_debug:
    type: "console"
    inputs: ["format_*"]
    encoding:
      codec: "json"

Vector Logs

2025-09-17T10:17:53.008157Z  INFO vector::app: Log level is enabled. level="info"
2025-09-17T10:17:53.009083Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.yaml"]
2025-09-17T10:17:53.014525Z  INFO source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector::sources::host_metrics: PROCFS_ROOT is set in envvars. Using custom for procfs. custom="/host/proc"
2025-09-17T10:17:53.014540Z  INFO source{component_kind="source" component_id=host_metrics component_type=host_metrics}: vector::sources::host_metrics: SYSFS_ROOT is set in envvars. Using custom for sysfs. custom="/host/sys"
2025-09-17T10:17:53.019377Z  WARN transform{component_kind="transform" component_id=format_solver_metrics component_type=remap}: vector::transforms::remap: VRL compilation warning. warnings=
warning[E900]: unused variable `err`
  ┌─ :2:13
  │
2 │ .timestamp, err = to_string(del(.timestamp))
  │             ^^^ help: use the result of this expression or remove it
  │
  = this expression has no side-effects
  = see language documentation at [https://vrl.dev⁠](https://vrl.dev/)
  = try your code in the VRL REPL, learn more at [https://vrl.dev/examples⁠](https://vrl.dev/examples)
2025-09-17T10:17:53.036854Z  INFO vector::topology::running: Running healthchecks.
2025-09-17T10:17:53.036913Z  INFO vector: Vector has started. debug="false" version="0.49.0" arch="aarch64" revision="dc7e792 2025-08-12 13:47:08.632326804"
2025-09-17T10:17:53.037073Z  INFO vector::topology::builder: Healthcheck passed.
2025-09-17T10:17:53.037777Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground graphql=http://0.0.0.0:8686/graphql
2025-09-17T10:17:53.038886Z  INFO vector::topology::builder: Healthcheck passed.
2025-09-17T10:17:53.038941Z  INFO vector::topology::builder: Healthcheck passed.
{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039721303Z","kind":"absolute","counter":{"value":106413.41}}
{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039734178Z","kind":"absolute","counter":{"value":11.29}}
{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039737594Z","kind":"absolute","counter":{"value":0.0}}
{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039740469Z","kind":"absolute","counter":{"value":112.59}}
{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039743219Z","kind":"absolute","counter":{"value":290.69}}
{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039746178Z","kind":"absolute","counter":{"value":107061.48}}
{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039748969Z","kind":"absolute","counter":{"value":12.95}}

and after that console sink just normally logs the original, non altered messages

[pront]

Hi @olekspickle,

For some reason my VRL transform works on [vrl playground] but not locally.

When using VRL playground, the input event is specified by the user and it is static. However, events passing through a real pipeline can have many different formats. So different behavior is expected.

but no data in clickhouse, and I see normal unaltered messages being logged into console.

Do you see any error logs? Did you also inspect the component_errors_total for clickhouse_host_metrics? (how to inspect internal metrics)

The console sink is printing the following metric:

{"name":"cpu_seconds_total","timestamp":"2025-09-17T10:17:53.039721303Z","kind":"absolute","counter":{"value":106413.41}}

which should be ingestible by Clickhouse.

I noticed a mistake in your config, .metric_name is an invalid metric path per https://vector.dev/docs/reference/configuration/transforms/remap/#event-data-model. That assignment will be ignored.

[olekspickle]

Huh, for some reason I thought I could just treat it as a json and remap it anyhow I want.
I do not have any component_errors_total after enabling internal metrics, just a lot of debug info but nothing that say that the insert to clickhouse failed

clickhouse query log does not have any inserts

In the end all I want to unify the metrics for clickhouse, but unable to remove the fields like counter,gauge etc which can lead to errors on insert. Can this be a silent failing point?
That would require to have separate tables for these. I did not find any good approach on this, is there a better way to deliver this to grafana via vector?
I mean these are the types metrics come from vector statsd/host source:

{"name":"source.response.success","tags":{"metric_type":"response"},"timestamp":"2025-09-18T10:00:29.845543012Z","kind":"absolute","gauge":{"value":0.0}}
{"name":"jsource.response.success","tags":{"metric_type":"response"},"timestamp":"2025-09-18T10:00:29.845567470Z","kind":"incremental","counter":{"value":1.0}}
{"name":"source.response.time_ms","tags":{"metric_type":"response"},"timestamp":"2025-09-18T10:00:30.478493304Z","kind":"incremental","distribution":{"samples":[{"value":632.0,"rate":1}],"statistic":"histogram"}}

not sure how am I supposed to not be able to delete the unexpected fields like distribution\counter\gauge and represent them in sql, lets say with such schema:

CREATE TABLE IF NOT EXISTS metrics.host (
    timestamp DateTime64(3) CODEC(Delta, LZ4),
    name LowCardinality(String),
    value Float64 CODEC(Gorilla, LZ4),
    source LowCardinality(String),
    tags Map(String, String) CODEC(ZSTD)
) ENGINE = MergeTree()
PARTITION BY (name, toYYYYMMDD(timestamp))  -- Partition by metric name + date for better pruning
ORDER BY timestamp
TTL timestamp + INTERVAL 90 DAY
SETTINGS index_granularity = 8192,
         merge_with_ttl_timeout = 86400;  -- Daily cleanup for better performance

how do you insert the metric data in clickhouse?..

[pront]

From the sink config I see that clickhosue sink only accepts logs

Good point. So basically, remap accepts all event types (logs, metrics, traces) and preserves the event type. Looking at your configuration, the clickhouse sink will never receive any events because all the events are metrics (see docs). You can use the metric_to_log transform though.

[olekspickle]

oh, so you have specific transform for this case? because I tried doing log(.) at the end and it did not work

[olekspickle]

also I think you meant metric_to_log, right?
and I would plug it right before my remap, correct?

[pront]

Yes to both!

[olekspickle]

ok, I finally see some data in clickhouse! now I only have to tune grafana queries and data source.

thank you for the help!