Vector Pipeline Editor

A visual drag-and-drop editor for creating and editing Vector.dev configuration files. Build data pipelines visually by connecting Vector components without writing TOML manually.

Features

Visual Pipeline Builder

• Drag & Drop Interface: Create pipelines by dragging components from palette to canvas
• Visual Connections: Connect components with visual lines showing data flow
• Component Types: Support for Sources, Transforms, and Sinks
• Route Handling: Special support for route transforms with multiple outputs

Configuration Management

• Text Editor: Monaco Editor with JSON syntax highlighting for component configuration
• Real-time Validation: Immediate feedback on configuration errors
• Component Templates: Pre-filled configurations for common use cases

File Operations

• Import TOML: Parse existing Vector configuration files
• Export TOML: Generate Vector-compatible TOML from visual pipeline
• Demo Mode: Load complex example pipeline to explore features

Supported Vector Components

Sources

• HTTP Server - Receive data via HTTP requests
• Socket - TCP/UDP socket listener
• File - Read from file system
• Internal Metrics - Vector internal metrics
• Kafka - Kafka consumer
• Syslog - Syslog receiver

Transforms

• Remap - VRL script processing
• Filter - Event filtering
• Route - Conditional routing with multiple outputs
• Sample - Event sampling
• JSON Parser - Parse JSON from fields
• Lua - Lua script processing

Sinks

• Loki - Send to Grafana Loki
• Prometheus - Export metrics
• Socket - TCP/UDP output
• File - Write to files
• Elasticsearch - Send to Elasticsearch
• Kafka - Kafka producer

Getting Started

Prerequisites

• Node.js 18+
• npm or yarn

Installation

# Install dependencies
npm install

# Start development server
npm run dev

Usage

1. Open the Editor: Navigate to http://localhost:5173

2. Create Components:

   • Drag components from the left palette to the canvas
   • Or click components to add them automatically

3. Connect Components:

   • Drag from output handles (right side) to input handles (left side)
   • Route transforms show multiple named outputs

4. Configure Components:

   • Click any component to select it
   • Edit configuration in the right panel using the JSON editor
   • Update component name as needed

5. Export Pipeline:

   • Click "Export TOML" to download Vector configuration
   • Use with Vector CLI: vector --config vector.toml

Demo Pipeline

Click "Load Demo" to see a complex example featuring:

• Fluent-operator input processing
• Debug pipeline with multiple filter stages
• Route transforms with conditional outputs
• Multiple Loki sinks with different configurations

Development

Project Structure

src/
├── components/
│   ├── blocks/          # Vector component blocks
│   ├── canvas/          # React Flow canvas
│   ├── editors/         # Configuration editors  
│   └── palette/         # Component palette
├── data/                # Component definitions
├── stores/              # Zustand state management
├── types/               # TypeScript definitions
└── utils/               # TOML parsing utilities

Building for Production

npm run build
npm run preview  # Preview production build

Adding New Components

1. Add definition to src/data/componentDefinitions.ts
2. Update types in src/types/vector.ts
3. Component automatically appears in palette

Architecture

• React Flow: Handles drag-and-drop canvas and visual connections
• Monaco Editor: Provides syntax-highlighted JSON editing
• Zustand: Manages application state and React Flow synchronization
• Tailwind CSS: Responsive styling and component design
• @iarna/toml: TOML parsing and generation

Vector.dev Integration

The editor generates standard Vector TOML configurations:

• Components become [sources.*], [transforms.*], [sinks.*] sections
• Visual connections become inputs arrays
• Route outputs use dot notation (e.g., route_name.output_name)

License

MIT License