valka

Installation & Configuration

Install Valka, choose where it stores its data, and configure it.

Valka is one binary, valka-server, plus one place to store data. There is no database to set up and no migrations to run: the first node to start on an empty bucket creates what it needs.

Storage backends

Everything durable lives in the storage backend: the write-ahead log, snapshots and task logs.

BackendUse it forNotes
localA single node, developmentA directory (default ./data). Single node only
s3Production, and any clusterAny S3-compatible bucket with conditional writes: AWS S3, MinIO, ...
memoryTests and demosLost when the process exits

The default is local. A cluster of several nodes needs the s3 backend; see Clustering.

The repository's docker-compose.yml starts an S3-compatible store and one Valka node:

git clone https://github.com/IWhitebird/Valka.git && cd Valka
docker compose up -d
ServiceDefault PortDescription
HTTP (REST + Dashboard)8989REST API, SSE events, Prometheus metrics, web dashboard
gRPC50051Worker connections, client SDKs, event and log streams
Object store9000, 9001S3 API and web console (minioadmin / minioadmin)

The store is pgsty/silo, an S3-compatible MinIO fork. A one-shot minio-init service creates the valka bucket. For production, point Valka at a real bucket and drop the store services.

Docker Image

You can also run the server image directly. With no settings it uses the local backend under /var/lib/valka/data, so mount a volume there:

docker run -d \
  -p 8989:8989 \
  -p 50051:50051 \
  -v valka-data:/var/lib/valka \
  ghcr.io/iwhitebird/valka:latest

Or point it at a bucket:

docker run -d \
  -e VALKA_NODE_ID=valka-1 \
  -e VALKA_STORAGE__BACKEND=s3 \
  -e VALKA_STORAGE__BUCKET=my-valka-bucket \
  -e VALKA_STORAGE__REGION=us-east-1 \
  -p 8989:8989 \
  -p 50051:50051 \
  ghcr.io/iwhitebird/valka:latest

The image includes the Valka server, the valka CLI and the web dashboard. It runs as a non-root user (UID 1001), includes curl for health checks and has a built-in HEALTHCHECK on /healthz. It exposes 50051 (gRPC), 8989 (REST and dashboard) and 50052 (node-to-node, used only by clusters; never expose it publicly).

From Source

cargo build --release -p valka-server -p valka-cli
./target/release/valka-server            # optionally: a path to valka.toml

Building needs Rust 1.89+ and the protobuf compiler (protoc). The server serves the dashboard from web/dist by default; build it with cd web && npm install && npm run build.

Configuration

Valka uses layered configuration via Figment:

Priority (highest wins): environment variables > valka.toml > defaults

Environment Variables

VariableDefaultDescription
VALKA_STORAGE__BACKENDlocals3, local (a directory) or memory
VALKA_STORAGE__PATH./dataDirectory for the local backend
VALKA_STORAGE__BUCKETvalkaBucket for the s3 backend
VALKA_STORAGE__ENDPOINT-Custom S3 endpoint (MinIO and others). Omit for AWS
VALKA_STORAGE__ACCESS_KEY_ID, VALKA_STORAGE__SECRET_ACCESS_KEY-Credentials. Unset: the AWS default chain (environment, profile, instance metadata)
VALKA_NODE_IDhost nameStable node id. A restart with the same id splits what the previous run left, so its shards are served again at once
VALKA_WAL__FLUSH_INTERVAL_MS50Group-commit window: how long a write may wait before its batch is written
VALKA_GRPC_ADDR0.0.0.0:50051gRPC listen address
VALKA_HTTP_ADDR0.0.0.0:8989REST/HTTP listen address
RUST_LOGvalka=info,tower_http=infoLog level filter

All configuration values can be set via environment variables with the VALKA_ prefix and __ for nested keys. See the full configuration reference for every option, and Clustering for the cluster settings.

Configuration File

Pass the path of a valka.toml as the first argument. With Docker, mount it into the container:

docker run -d \
  -v ./valka.toml:/etc/valka/valka.toml:ro \
  -p 8989:8989 \
  -p 50051:50051 \
  ghcr.io/iwhitebird/valka:latest \
  /etc/valka/valka.toml
valka.toml
node_id = "valka-1"
grpc_addr = "0.0.0.0:50051"
http_addr = "0.0.0.0:8989"

[storage]
backend = "s3"
bucket = "valka"
# endpoint = "http://minio:9000"   # S3-compatible stores; omit for AWS
# region = "us-east-1"
# allow_http = true                # plain-HTTP endpoints, development only

[wal]
flush_interval_ms = 50

[scheduler]
retry_base_delay_secs = 1
retry_max_delay_secs = 3600

deploy/valka.example.toml in the repository lists every setting with its default.

What Valka Writes

The first node to start on an empty bucket writes cluster.json, which fixes the shard count (default 4096) and the storage format. After that the bucket holds:

KeyHolds
nodes/{node}/leaseEach node's lease: who is alive
owners/{hh}/{shard}Which node owns each shard
queues/{hh}/{queue}Each queue's partition count
wal/{node}/...Each node's write-ahead log segments
snapshots/{hh}/{shard}/...Per-shard state checkpoints
splits/..., placement.json, admin/...Takeover markers, operator pins, admin audit records
logs/{hh}/{run}/...Task log lines

Use one bucket, or one prefix inside a bucket (VALKA_STORAGE__PREFIX), per Valka cluster. See Architecture for what each object does.

Health Check

The server exposes a health endpoint for container orchestrators:

curl http://localhost:8989/healthz
# Returns: "ok" (200), or "wal writer poisoned" (503) just before the node exits

On this page