valka

Deployment

Run Valka in production. One node or a cluster on Docker Compose, VMs or Kubernetes, with health checks, alerts and the full configuration reference.

A Valka deployment is one or more valka-server processes and one bucket. Nodes keep nothing on their own disks that the bucket doesn't also have, so you can replace, restart or kill them at any time. Run every node under a supervisor that restarts it when it exits (Docker restart: unless-stopped, systemd, Kubernetes): a node exits on purpose when it can't write to the bucket or finds itself fenced, and a fresh process rebuilds from the bucket.

PortServesExpose to
50051Public gRPC: workers, SDK clients, CLIWorkers and clients
8989REST API, dashboard, SSE events, /healthz, /metricsClients, operators, Prometheus
50052Internal node-to-node gRPC (cluster only)Other Valka nodes only. Never public

The public REST and gRPC APIs have no authentication of their own (only the admin endpoints need a token). Keep them on a private network or behind an authenticating proxy.


Single node

A single node can use any storage backend. It runs the same protocol as a cluster, alone.

With a local directory

The simplest setup: the local backend stores everything in one directory. The Docker image defaults to it, under /var/lib/valka/data.

docker run -d --name valka --restart unless-stopped \
  -p 8989:8989 -p 50051:50051 \
  -v valka-data:/var/lib/valka \
  ghcr.io/iwhitebird/valka:latest

Or run the binary directly; it uses ./data by default:

valka-server            # or: valka-server /etc/valka/valka.toml

That directory is the bucket: the WAL, snapshots and task logs live there. Back it up like a database, and never point two processes at it. The local backend is single-node only; a cluster needs an S3-compatible bucket.

With an S3-compatible bucket

Point the node at a bucket with the s3 backend:

docker run -d --name valka --restart unless-stopped \
  -p 8989:8989 -p 50051:50051 \
  -e VALKA_NODE_ID=valka-1 \
  -e VALKA_STORAGE__BACKEND=s3 \
  -e VALKA_STORAGE__BUCKET=my-valka-bucket \
  -e VALKA_STORAGE__REGION=us-east-1 \
  ghcr.io/iwhitebird/valka:latest

Credentials come from VALKA_STORAGE__ACCESS_KEY_ID and VALKA_STORAGE__SECRET_ACCESS_KEY, or, when those are unset, from the AWS default chain (environment, profile, instance metadata, IRSA). For MinIO and other S3-compatible stores, also set VALKA_STORAGE__ENDPOINT (and VALKA_STORAGE__ALLOW_HTTP=true for a plain-HTTP endpoint in development).

Set a fixed VALKA_NODE_ID. It defaults to the host name, which changes with every new container unless you set a hostname. A restarted node with the same id fences its previous run, splits what that run left in the log and reclaims its shards at once.

With Docker Compose

The repository's docker-compose.yml starts an S3-compatible store and one node built from source:

git clone https://github.com/IWhitebird/Valka.git && cd Valka
docker compose up -d
ServicePortDescription
valka-server8989, 50051REST, dashboard, gRPC. Node id valka-1
minio9000, 9001S3 API and web console (minioadmin / minioadmin)
minio-init-Creates the valka and valka-test buckets, then exits

The store is pgsty/silo, an S3-compatible MinIO fork (the minio/minio images are no longer published on Docker Hub). In production, point [storage] at a real bucket and drop the store services. deploy/docker/docker-compose.yml is a variant of the same setup whose bucket, endpoint and credentials can be overridden with environment variables (S3_BUCKET, S3_ENDPOINT, S3_ACCESS_KEY_ID, ...).


A three-node cluster with Docker Compose

docker-compose.cluster.yml runs three nodes on one S3-compatible store:

docker compose -f docker-compose.cluster.yml up -d --build
NodeREST API and dashboardgRPC
valka-1http://localhost:8981localhost:50061
valka-2http://localhost:8982localhost:50062
valka-3http://localhost:8983localhost:50063

Every node gets the same storage settings, VALKA_CLUSTER__ENABLED=true, the same cluster token and an admin token, plus its own VALKA_NODE_ID and advertised addresses. Change both tokens before using it beyond a laptop. The build is limited to 4 parallel compile jobs to keep a laptop usable; it takes a while the first time.

See Clustering to watch a takeover on it.


VMs and bare metal

Run one valka-server per machine, all pointing at the same bucket. Every node needs:

  • the same [storage] settings (bucket, prefix, endpoint, region);
  • VALKA_CLUSTER__ENABLED=true and the same VALKA_CLUSTER__TOKEN;
  • a distinct, stable VALKA_NODE_ID;
  • advertised addresses the other nodes and clients can reach.
/etc/valka/valka.env (node 1 of 3)
VALKA_NODE_ID=valka-1
VALKA_STORAGE__BACKEND=s3
VALKA_STORAGE__BUCKET=my-valka-bucket
VALKA_STORAGE__REGION=eu-west-1
VALKA_CLUSTER__ENABLED=true
VALKA_CLUSTER__TOKEN=<a long random secret, the same on every node>
VALKA_CLUSTER__INTERNAL_ADDR=0.0.0.0:50052
VALKA_CLUSTER__ADVERTISE_INTERNAL_ADDR=10.0.0.1:50052
VALKA_CLUSTER__ADVERTISE_GRPC_ADDR=10.0.0.1:50051
VALKA_CLUSTER__ADVERTISE_HTTP_ADDR=10.0.0.1:8989
VALKA_API_KEYS=<keys for producers and workers, comma-separated>
VALKA_ADMIN_TOKEN=<another secret, only if you want the admin API>

Nodes 2 and 3 differ only in VALKA_NODE_ID and their addresses. Open 50052 between the nodes and nowhere else. For mTLS on the internal port, see Clustering.

Give workers and clients every node's address (all four SDKs and the CLI take a list), or a load balancer or DNS name in front of all of them, so a worker that loses its node reconnects to another one and a create that hits a dying node is retried on the next. Clients send one of the API keys (VALKA_API_KEY in the SDKs and the CLI).

No systemd units ship yet. Any supervisor works, as long as it restarts the process when it exits and gives it time to stop (see Graceful shutdown).

Bucket requirements

The protocol relies on the bucket's consistency, so a cluster needs a store with:

  • conditional writes: create-if-absent PUT and If-Match PUT;
  • strong read-after-write and strong LIST consistency;
  • stable ETags.

AWS S3 and recent MinIO provide these. A clustered node runs a storage self-test at start: it creates a probe object twice (the second create must fail), CASes with a wrong ETag (must fail) and with the right one (must succeed), and checks that LIST shows the probe at once. If any check fails, the node refuses to start.

Use one bucket, or one prefix inside a bucket, per cluster. The first node writes cluster.json with the shard count and the storage format; every later node must agree with it.

Capacity

A node holds the live tasks of the shards it owns in RAM, plus finished tasks for completed_retention_secs (24 h by default); their history stays in the bucket. Adding a node adds capacity: placement moves about 1/N of the shards to it after the join delay. Useful nodes top out around num_shards / 8, so choose the shard count of a new bucket with growth in mind.


Kubernetes

The Helm chart in deploy/helm predates the WAL architecture and is being reworked. Don't use it with this version. Until the new chart lands, run the container yourself as a StatefulSet-like deployment, or use Docker Compose or VMs:

  • one pod per node, with a stable node id: in a StatefulSet the pod name is the host name, so the default node id is already stable;
  • a headless service, with VALKA_CLUSTER__ADVERTISE_INTERNAL_ADDR set to each pod's stable DNS name and port 50052;
  • the same storage settings and cluster token for every pod (the token from a Secret);
  • no volume needed with the s3 backend;
  • a termination grace period long enough for the node to hand its shards over on SIGTERM;
  • liveness and readiness probes on GET /healthz (port 8989);
  • port 50052 reachable only from the other pods.

Chaos test

scripts/chaos.sh breaks the three-node compose cluster while a load generator runs:

scripts/chaos.sh                     # 5 minutes of chaos
DURATION=1800 scripts/chaos.sh       # a 30-minute soak
NO_BUILD=1 scripts/chaos.sh          # reuse the valka:local image

The load generator (examples/rs/chaos.rs) creates tasks through every node (20 per second by default, RATE), runs a worker on every node and records every acknowledged create. Meanwhile the script picks a random node every 10–30 s and either kill -9s it, freezes it (SIGSTOP, like a long GC pause or a VM freeze) or restarts it with SIGTERM. At the end every node comes back, and the checker verifies that every acknowledged task exists and finished, and that no idempotency key produced two tasks. It needs Docker Compose and cargo, writes its logs to target/chaos, and exits 0 when every check passes.


Monitoring and alerts

Health checks

curl http://localhost:8989/healthz   # "ok" (200), or 503 "wal writer poisoned"
curl http://localhost:8989/metrics   # Prometheus text format

/healthz answers 503 only when the node's WAL writer is poisoned, and such a node exits right after. It does not report a self-fenced node (one that can't renew its lease); watch the valka_node_serving metric for that. The Docker image has a built-in HEALTHCHECK on /healthz.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8989
  initialDelaySeconds: 15
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /healthz
    port: 8989
  periodSeconds: 5

For a cluster-wide view, GET /api/v1/cluster on any node returns every node, the number of unowned shards and an overall health.status (ok, degraded or critical).

Metrics

Every node exports /metrics on its HTTP port. The cluster metrics:

MetricMeaning
valka_node_serving1 while the node's lease is current, 0 when it self-fenced
valka_lease_age_secondsTime since the last successful lease renewal
valka_shards_ownedShards this node serves
valka_cluster_membersNodes this node sees
valka_node_fenced_totalTimes this node found its lease taken
valka_tombstones_totalNodes declared dead
valka_splits_total, valka_split_failures_total, valka_split_duration_secondsLog splits
valka_shard_claims_total, valka_shard_releases_totalShard moves
valka_forwarded_commands_totalRequests forwarded to other nodes, by result
valka_route_redirects_totalNOT_OWNER redirects followed
valka_reattach_totalWorker runs reattached after a reconnect
valka_wal_segments_total, valka_wal_bytes_totalWAL segments and bytes written

Task metrics include valka_tasks_created_total, valka_tasks_completed_total, valka_tasks_failed_total, valka_tasks_retried_total, valka_tasks_dead_lettered_total, valka_pending_tasks, valka_active_workers and valka_task_duration_ms.

Alert rules

deploy/prometheus/alerts.yml has ready-made Prometheus rules. Each one names a section of the runbooks, which say what to do when it fires.

AlertFires whenRunbook
ValkaNodeNotServing (critical)A node self-fenced for 30 sNode not serving
ValkaLeaseStaleLease renewals take over 6 s for 1 minBucket slow or down
ValkaNodeFencedA node found its lease takenNode fenced
ValkaTakeoverA node was declared dead and taken overNode lost
ValkaSplitFailing (critical)A dead node's log split keeps failingStuck split
ValkaUnownedShards (critical)Fewer shards served than exist, for 2 minUnowned shards
ValkaClaimInvariant (critical)A claim found the bucket inconsistent with the protocolClaim invariant
ValkaForwardingErrorsNodes can't reach each other's internal listenerNodes cannot reach each other

ValkaUnownedShards compares against 4096. Change the number if your cluster was created with another shard count.


Graceful shutdown

Stopping a server (SIGTERM or Ctrl-C):

  1. Every connected worker receives ServerShutdown and its stream is closed, so the server never waits on open streams.
  2. Workers keep running their in-flight tasks and reconnect with backoff, to another node or to the restarted one. Results that finish in the meantime are kept and delivered after the reconnect.
  3. If live peers exist, the node drains: it marks its lease draining, releases every shard with a final snapshot, marks its lease dead and exits. The peers claim the shards with nothing to replay.
  4. If it is alone, it keeps its shards: it flushes the WAL, snapshots changed shards and exits. The restart reclaims them with little to replay.

Give the process time to finish. Docker's default stop timeout is 10 s; the chaos script uses docker compose restart -t 30.

Rolling restart. Restart one node at a time, and wait until GET /api/v1/cluster shows no unowned shards before the next one. Each SIGTERM hands that node's shards to the others.

Drain without a restart. POST /api/v1/cluster/nodes/{node}/drain (admin API) makes a node hand everything over and exit.

Stopping a worker:

  1. The worker sends GracefulShutdown: the server stops sending it tasks but keeps the stream open.
  2. In-flight handlers finish and their results are acknowledged (the worker waits up to 5 s for the acks).
  3. The worker closes the stream and exits.

When the bucket fails

  • Writes are acknowledged only after their WAL segment is in the bucket, so a slow bucket means slower acknowledgements and, eventually, UNAVAILABLE. Acknowledged work is never lost.
  • A node that can't renew its lease for 7 s stops serving until it can. If it stays cut off for 10 s, another node takes its shards over.
  • If WAL PUTs keep failing after their retries (put_retries, default 8), the node fails the pending writes with UNAVAILABLE and exits with a non-zero status. Its supervisor restarts it, and it rebuilds its state from the bucket.
  • If the bucket is down for everyone, every node stops serving and nobody can declare anybody dead. The cluster recovers by itself when the bucket returns.

Upgrades

Restart nodes one at a time onto the new build, as in a rolling restart. Each lease advertises the node's protocol version, and the cluster's effective protocol is the lowest among live nodes (effective_protocol in GET /api/v1/cluster/placement). A node refuses to join a cluster whose live nodes already speak a newer protocol.


Configuration reference

Configuration is layered: built-in defaults, then a TOML file, then VALKA_* environment variables (highest wins). Nested keys use a double underscore: [storage] bucket is VALKA_STORAGE__BUCKET. Pass the config file path as the first argument: valka-server /etc/valka/valka.toml.

Environment variables

VariableDefaultDescription
VALKA_NODE_IDhost nameStable node id (1–64 characters from A-Z a-z 0-9 _ -)
VALKA_ADMIN_TOKEN-Enables the admin API
VALKA_API_KEYS-API keys for the public API, comma-separated. Unset: the API is open (the server warns)
VALKA_LIMITS__MAX_TASKS_IN_RAM1000000Tasks a node holds in RAM; above it creates get 429
VALKA_LIMITS__MAX_PAYLOAD_BYTES1048576Largest task input or metadata
VALKA_GRPC_ADDR0.0.0.0:50051gRPC listen address
VALKA_HTTP_ADDR0.0.0.0:8989REST, dashboard and metrics listen address
VALKA_WEB_DIRweb/distBuilt dashboard files (the image sets /usr/share/valka/web)
VALKA_STORAGE__BACKENDlocals3, local (a directory) or memory (tests only)
VALKA_STORAGE__PATH./dataDirectory for the local backend
VALKA_STORAGE__BUCKETvalkaBucket for the s3 backend
VALKA_STORAGE__PREFIXemptyKey prefix inside the bucket, e.g. valka/prod
VALKA_STORAGE__ENDPOINT-Custom S3 endpoint (MinIO, R2, ...). Omit for AWS
VALKA_STORAGE__REGION-Bucket region
VALKA_STORAGE__ACCESS_KEY_ID, VALKA_STORAGE__SECRET_ACCESS_KEY-Credentials. Unset: the AWS default chain
VALKA_STORAGE__ALLOW_HTTPfalseAllow plain-HTTP endpoints (development only)
VALKA_WAL__FLUSH_INTERVAL_MS50Group-commit window: acknowledgement latency vs. PUT count
VALKA_WAL__MAX_BATCH_BYTES4194304Flush early at this many buffered bytes
VALKA_WAL__MAX_INFLIGHT_SEGMENTS4Segment PUTs in flight at once
VALKA_WAL__PUT_RETRIES8PUT retries before writes fail and the node exits
VALKA_WAL__SNAPSHOT_INTERVAL_SECS60Snapshot changed shards at least this often
VALKA_WAL__SNAPSHOT_AFTER_RECORDS50000...or once a shard has this many new records
VALKA_WAL__LOG_BUDGET_BYTES67108864...or once this much log is uncovered. Bounds restart and takeover time
VALKA_WAL__SNAPSHOTS_TO_KEEP2Snapshots kept per shard
VALKA_WAL__COMPLETED_RETENTION_SECS86400Finished tasks leave RAM after this (their history stays in the bucket)
VALKA_CLUSTER__*Cluster settings: see Clustering
VALKA_SCHEDULER__TIMER_TICK_MS100Timer resolution for lease expiry, retries and delayed tasks
VALKA_SCHEDULER__LEASE_GRACE_SECS30Added to a task's timeout for its first lease
VALKA_SCHEDULER__HEARTBEAT_LEASE_SECS60Each heartbeat keeps the lease at least this far ahead
VALKA_SCHEDULER__RECOVERY_GRACE_SECS60Lease given to recovered RUNNING tasks until their worker reconnects
VALKA_SCHEDULER__RETRY_BASE_DELAY_SECS1Retry backoff base
VALKA_SCHEDULER__RETRY_MAX_DELAY_SECS3600Retry backoff cap
RUST_LOGvalka=info,tower_http=infoLog filter

Full configuration file

Every setting with its default and a comment, from deploy/valka.example.toml:

valka.toml
# Valka Server Configuration Reference
#
# Config layering: defaults -> this file -> VALKA_* env vars
# Nested env vars use double underscore: VALKA_STORAGE__BUCKET=valka
# Pass config path as first CLI argument: valka-server /etc/valka/valka.toml

# Unique, stable node identifier. Defaults to the host name if empty.
# A restarted node with the same id fences its previous run, splits what that run left in
# wal/<node_id>/ and reclaims its shards. Every node of a cluster needs a distinct id.
node_id = ""

# Bearer token for the admin API (drain a node, move a shard, set queue partitions,
# audit log). Unset = admin API disabled. Env: VALKA_ADMIN_TOKEN.
# admin_token = ""

# API keys for the public REST and gRPC APIs, comma-separated (env: VALKA_API_KEYS).
# Clients send one as "Authorization: Bearer <key>"; the SDKs and the CLI read
# VALKA_API_KEY. Unset = the public API is OPEN to anyone who can reach its ports (the
# server logs a warning). /healthz and /metrics stay open.
# api_keys = "key-for-producers,key-for-workers"

# gRPC listen address (workers and API clients)
grpc_addr = "0.0.0.0:50051"

# HTTP/REST + WebUI listen address
http_addr = "0.0.0.0:8989"

# Path to built WebUI static files
web_dir = "/usr/share/valka/web"

# --- Storage (the only durable state) --------------------------------------

[storage]
# "s3"     any S3-compatible bucket (AWS S3, MinIO, R2, ...). Needs conditional writes.
# "local"  a directory (dev / single machine). CAS is emulated in-process.
# "memory" ephemeral, tests and demos only.
backend = "s3"

# local backend: directory
path = "./data"

# s3 backend
bucket = "valka"
# Optional key prefix inside the bucket, e.g. "valka/prod"
prefix = ""
# Custom endpoint for S3-compatible stores (MinIO: "http://minio:9000"). Omit for AWS.
# endpoint = "http://minio:9000"
# region = "us-east-1"
# Credentials. Omit to use the AWS default chain (env vars, profile, IMDS / IRSA).
# access_key_id = ""
# secret_access_key = ""
# Allow plain HTTP endpoints (MinIO in dev only)
allow_http = false

# --- Write-ahead log -------------------------------------------------------

[wal]
# Group-commit window. Nothing is acknowledged before its segment is in the bucket.
# Lower = lower enqueue latency, more PUT requests. 50-100 ms is the sweet spot on S3.
flush_interval_ms = 50
# Flush early once the buffer reaches this many bytes.
max_batch_bytes = 4194304
# Segment PUTs in flight concurrently.
max_inflight_segments = 4
# PUT retries before acks fail with UNAVAILABLE (the node then exits and restarts).
put_retries = 8
# Snapshot dirty shards at least this often; segments below every snapshot are deleted.
snapshot_interval_secs = 60
# ...or as soon as a shard has this many unsnapshotted records.
snapshot_after_records = 50000
# ...or, for every dirty shard, once this many WAL bytes were written since the last
# full round. Bounds restart replay time. 0 disables.
log_budget_bytes = 67108864
# Snapshots kept per shard.
snapshots_to_keep = 2
# Terminal tasks (COMPLETED/FAILED/CANCELLED/DEAD_LETTER) leave RAM after this.
# Their history stays in the bucket.
completed_retention_secs = 86400

# --- Cluster ---------------------------------------------------------------
# One node is a cluster of one: the same lease / ownership protocol always runs.
# See docs/ARCHITECTURE.md §8 and docs/wal/PHASE2.md.

[cluster]
# Allow other nodes on this bucket. Needs the s3 backend (local/memory CAS only works
# inside one process) and a token.
enabled = false

# Shared secret every node-to-node call carries. Required when enabled.
# token = ""

# Shard count of a NEW bucket (written to cluster.json on first start; fixed afterwards).
# Power of two, 16..=65536. 0 = whatever the bucket says, or 4096 for a new bucket.
num_shards = 0

# Node-to-node listener. Never expose it publicly.
internal_addr = "0.0.0.0:50052"
# Addresses other nodes / clients use to reach this node (default: the listen addresses).
# advertise_internal_addr = "10.0.0.1:50052"
# advertise_grpc_addr = "10.0.0.1:50051"
# advertise_http_addr = "10.0.0.1:8989"

# Optional mTLS for internal traffic (PEM files). The token is still required.
# tls_cert = "/etc/valka/tls/node.crt"
# tls_key = "/etc/valka/tls/node.key"
# tls_ca = "/etc/valka/tls/ca.crt"

# Lease renewal period, and how long an unchanged lease takes to be declared dead
# (by the observer's own clock). A node that has not renewed for
# lease_timeout - lease_safety_margin stops serving on its own.
lease_interval_ms = 2000
lease_timeout_ms = 10000
lease_safety_margin_ms = 3000

# How often nodes/ is listed (membership).
membership_interval_ms = 3000
# A new node takes shards after this (all nodes at once on a cold start).
join_delay_ms = 15000
# Placement round period, claims in flight, shards handed over per round.
placement_interval_ms = 3000
claim_concurrency = 64
handover_per_tick = 64
# A shard stays where it is for at least this long after a move.
min_residency_secs = 300
# Another node may redo a dead node's log split after this.
split_timeout_ms = 30000
# After claiming shards, recovered PENDING tasks wait this long so reconnecting workers
# can reattach their runs first.
reconcile_grace_ms = 5000
# Garbage collector cadence (dead logs, old snapshots, split markers).
gc_interval_secs = 300
# Refuse to start while another live process holds this node id.
startup_lease_check = true

# --- Limits ------------------------------------------------------------------

[limits]
# Tasks a node may hold in RAM (live, plus finished ones within completed_retention_secs).
# Above it, creates are refused with RESOURCE_EXHAUSTED / HTTP 429 until the backlog
# drains; near it, finished tasks leave RAM early (their history stays in the bucket).
# 0 disables the limit.
max_tasks_in_ram = 1000000
# Largest task input or metadata, in bytes of JSON. Store big data elsewhere and pass a
# reference.
max_payload_bytes = 1048576

# --- Matching / Task Routing -----------------------------------------------

[matching]
# In-RAM worker-routing partitions per queue on one node (unrelated to storage shards
# and to queue partitions, which live in the bucket: queues/{hh}/{queue}).
num_partitions = 4

# Internal tree branching factor (rarely needs tuning)
branching_factor = 3

# Max tasks buffered per partition before back-pressure
max_buffer_per_partition = 1000

# How often the feeder tops up matching buffers from the pending index (ms)
feeder_interval_ms = 20

# Max tasks moved per queue per feeder tick
feeder_batch_size = 200

# --- Scheduler -------------------------------------------------------------

[scheduler]
# Timer resolution for lease expiry, retry promotion and delayed tasks (ms)
timer_tick_ms = 100

# Extra time on top of a task's timeout before its first lease expires (seconds)
lease_grace_secs = 30

# Lease granted to RUNNING tasks recovered from the WAL until their worker reconnects
recovery_grace_secs = 60

# Lease granted by each heartbeat: the worker must heartbeat again within this window
heartbeat_lease_secs = 60

# Exponential backoff base for retries (seconds)
retry_base_delay_secs = 1

# Maximum retry delay cap (seconds)
retry_max_delay_secs = 3600

# --- Log Ingester ----------------------------------------------------------

[log_ingester]
# Max log lines buffered before a chunk is written to the bucket
batch_size = 100

# Max time to buffer logs before flushing (ms)
flush_interval_ms = 500

On this page