Quickwit

Quickwit is a cloud-native distributed search engine built in Rust, purpose-designed for observability workloads — logs, traces, and metrics. Its defining architectural choice is that it stores all index data on object storage (S3, GCS, Azure Blob) rather than local disk, decoupling compute from storage and enabling search costs that are typically 10× lower than Elasticsearch for the same workload.

It is built on top of Tantivy, a Rust full-text search library analogous to Apache Lucene, and exposes an Elasticsearch-compatible API so existing tooling (Grafana, Jaeger, OpenTelemetry collectors, Vector, Fluent Bit) works with little or no reconfiguration.

Features

• Object storage as primary storage — indexes live on S3 (or compatible), not on local disk; compute nodes are fully stateless and disposable
• Sub-second search on cold data — optimised IO paths and index data structures enable fast searches directly on object storage without a warm cache
• Elasticsearch-compatible API — existing log shippers, dashboards, and query clients connect without modification
• OTEL-native — first-class support for OpenTelemetry logs and traces; ingest directly from any OTEL collector
• Jaeger-native — implements the Jaeger gRPC API for distributed trace storage and querying
• Schemaless and strict-schema modes — ingest JSON without a schema, or define a doc mapping for performance and type safety
• Multi-tenancy — multiple indexes with independent retention policies and access controls
• Kafka / Kinesis / Pulsar sources — native connectors for durable, high-throughput ingestion
• Grafana datasource — official Grafana plugin for dashboards and alerting on Quickwit data
• Kubernetes-ready — official Helm chart for production deployment

Installation

# Debian / Ubuntu / Fedora (and all Linux distros)
curl -L https://install.quickwit.io | sh

# macOS (Homebrew)
brew install quickwit-oss/tap/quickwit

# cargo
cargo install quickwit

Or run via Docker:

docker pull quickwit/quickwit
docker run -p 7280:7280 quickwit/quickwit run

Quick Start

# Start a standalone server (uses local storage by default)
quickwit run

# The REST API is now available at http://localhost:7280/

# Create an index
curl -X POST http://localhost:7280/api/v1/indexes \
  -H 'Content-Type: application/yaml' \
  --data-binary @index-config.yaml

# Ingest NDJSON documents
curl -X POST http://localhost:7280/api/v1/my-index/ingest \
  -H 'Content-Type: application/json' \
  --data-binary @documents.ndjson

# Search
curl 'http://localhost:7280/api/v1/my-index/search?query=error&max_hits=10'

Index configuration

Quickwit index configuration is defined in YAML:

version: 0.7

index_id: my-logs

doc_mapping:
  mode: dynamic # accept any JSON fields
  field_mappings:
    - name: timestamp
      type: datetime
      fast: true
      input_formats: [unix_timestamp, rfc3339]
    - name: severity
      type: text
      tokenizer: raw # exact-match only
      fast: true
    - name: message
      type: text
      tokenizer: default

indexing_settings:
  timestamp_field: timestamp
  commit_timeout_secs: 10

search_settings:
  default_search_fields: [message]

retention:
  period: 30 days
  schedule: daily

Usage

# Run the server
quickwit run

# Run with an S3 backend
quickwit run --metastore-uri s3://my-bucket/quickwit-meta \
             --default-index-root-uri s3://my-bucket/quickwit-indexes

# Create an index from a config file
quickwit index create --index-config ./index-config.yaml

# Ingest a file
quickwit index ingest --index my-logs --input-path logs.ndjson

# Search from the CLI
quickwit index search --index my-logs --query "ERROR" --max-hits 20

# Describe an index (size, doc count, splits)
quickwit index describe --index my-logs

# List all indexes
quickwit index list

# Delete an index
quickwit index delete --index my-logs

Ingestion via OpenTelemetry

Any OTEL-compatible collector can send logs and traces directly to Quickwit:

# otel-collector-config.yaml
exporters:
  otlp:
    endpoint: "http://localhost:7281" # Quickwit OTLP gRPC endpoint
    tls:
      insecure: true

service:
  pipelines:
    logs:
      exporters: [otlp]
    traces:
      exporters: [otlp]

Architecture

Quickwit separates the roles of indexers (which write to object storage) and searchers (which read from it). Both are stateless:

• Indexers consume from sources (Kafka, OTEL, HTTP ingest), build index segments locally, and upload them to object storage
• Searchers download only the relevant segments for a query, execute the search, and return results
• Metastore tracks index metadata; backed by PostgreSQL, S3, or a local file

This means you can scale indexers and searchers independently, and a lost node never causes data loss.

Why Quickwit over Elasticsearch?

Elasticsearch stores indexes on local SSD attached to each node. At observability scale (terabytes of logs), this means expensive storage-heavy instances. Quickwit's object-storage-first design means you pay S3 prices for storage and only spin up compute when you need to search — typically 5–10× cheaper for equivalent query SLAs on cold log data.