How to read and transform .parquet , .gz files using vector.dev ? #24248
-
QuestionI'm trying to set up a pipeline to collect various access logs (CloudFront, Cloudflare, NLB, VPC Flow logs, etc.) from S3 using Vector. My goal is to either apply the logs_to_metrics transformer or transform the data before ingesting it into Grafana Loki. The challenge I'm facing is understanding how to read and parse .parquet files with Vector. While I can see functions like parse_json available in VRL (Vector Remap Language), I haven't been able to find a parse_parquet equivalent. Would I need to use a Lua script to handle Parquet parsing, or is there a better approach I'm overlooking? Any ideas or guidance would be greatly appreciated! Vector ConfigNo response Vector LogsNo response |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Vector.dev does not support encoding and decoding of Parquet files as of now. Lua is the only way if you want to use Vector.dev only. otherwise, you can go for Redpanda Connect (Benthos) as well. |
Beta Was this translation helpful? Give feedback.
-
|
Also, see #1374 |
Beta Was this translation helpful? Give feedback.
Vector.dev does not support encoding and decoding of Parquet files as of now. Lua is the only way if you want to use Vector.dev only. otherwise, you can go for Redpanda Connect (Benthos) as well.