Vector

Vector is a high-performance observability data pipeline written in Rust. It collects logs, metrics, and traces from any number of sources, applies configurable transformations, and routes the results to any number of destinations — all in a single binary with no runtime dependencies.

It is used to replace or complement heavy agents like Logstash or Fluentd, with dramatically lower resource consumption and higher throughput.

Features

• Unified pipeline — handles logs, metrics, and traces in a single tool using a consistent configuration model
• Rich source support — ingests from files, stdin, syslog, Kafka, Docker, Kubernetes, Prometheus, statsd, HTTP, AWS S3, GCP Pub/Sub, and many more
• Rich sink support — routes to Elasticsearch, ClickHouse, Datadog, Splunk, S3, GCS, BigQuery, Kafka, Loki, InfluxDB, Prometheus, HTTP endpoints, and many more
• Powerful transforms — filter, parse (regex, JSON, Grok, CSV, VRL), enrich, deduplicate, aggregate, sample, and route events using the built-in Vector Remap Language (VRL)
• Vector Remap Language (VRL) — a safe, fast, purpose-built scripting language for transforming observability data, with hundreds of built-in functions
• End-to-end acknowledgements — guarantees at-least-once delivery by tracking acknowledgements from sinks back to sources
• Topology — components connect as a directed acyclic graph; fan-out and fan-in are both first-class
• Hot reload — reload configuration without restarting or dropping events
• Built-in metrics — Vector instruments itself and exposes its own internal metrics for monitoring pipeline health
• Unit testing — test your VRL transforms with built-in unit test support in the config file

Installation

# macOS
brew install vector

# Debian/Ubuntu
curl -1sLf 'https://repositories.timber.io/public/vector/cfg/setup/bash.deb.sh' | bash
apt install vector

# Arch Linux
pacman -S vector

# via cargo
cargo install vector --locked --no-default-features --features default-musl

Or download a pre-built binary from the releases page.

Configuration

Vector is configured via a TOML (or YAML/JSON) file describing sources, transforms, and sinks as a pipeline graph:

# vector.toml

# Source: tail a log file
[sources.app_logs]
type   = "file"
include = ["/var/log/myapp/*.log"]

# Transform: parse JSON log lines and add a field
[transforms.parse_logs]
type   = "remap"
inputs = ["app_logs"]
source = '''
  . = parse_json!(string!(.message))
  .environment = "production"
'''

# Transform: filter out health check noise
[transforms.filter_health]
type   = "filter"
inputs = ["parse_logs"]
condition = '.path != "/health"'

# Sink: send to Loki
[sinks.loki]
type     = "loki"
inputs   = ["filter_health"]
endpoint = "http://localhost:3100"
labels   = { app = "myapp", env = "{{ environment }}" }

# Sink: also send to a local file for debugging
[sinks.debug_file]
type   = "file"
inputs = ["filter_health"]
path   = "/tmp/vector-debug-%Y-%m-%d.log"

Usage

# Validate a config file
vector validate --config vector.toml

# Run with a config file
vector --config vector.toml

# Run with hot reload on config change
vector --config vector.toml --watch-config

# Test VRL transforms defined in the config
vector test --config vector.toml

# Generate a config skeleton for a specific component
vector generate source/file sink/elasticsearch

# List all available components
vector list

Vector Remap Language (VRL)

VRL is the transform scripting language built into Vector. It is safe (no network access, no filesystem access), fast (compiles to bytecode), and purpose-built for log and metric manipulation:

# Parse an Apache Combined Log Format line
. = parse_apache_log!(.message, format: "combined")

# Redact sensitive fields
.email = redact(.email, filters: ["pattern"], redactor: "full", patterns: [r'\S+@\S+\.\S+'])

# Enrich with a static lookup table
.region = get!(value: {"us-east-1": "US East", "eu-west-1": "EU West"}, path: [.aws_region])

# Drop the event if it's below error level
if .level != "error" && .level != "warn" {
  abort
}

Why Vector over Logstash or Fluentd?

Vector consistently benchmarks at 10–100× the throughput of Logstash with a fraction of the memory usage. As a compiled Rust binary with no JVM or Ruby runtime overhead, it starts in milliseconds and idles at a few MB of RAM — making it practical even on resource-constrained edge nodes or sidecar containers.