Clustering
Run many Valka nodes on one bucket. Leases, shard ownership, routing, takeover and the admin API.
A Valka cluster is a set of valka-server processes that point at the same bucket. There
is no gossip, no leader election and no coordination service. Nodes learn who is alive and who
owns what from objects in the bucket, changed only by conditional writes. One node is simply a
cluster of one: it runs the same protocol, alone.
graph TD
C["Producers<br/>REST · gRPC · CLI"] --> N1
C --> N2
C --> N3
subgraph Cluster["Valka cluster"]
N1["valka-1<br/>owns about 1/3 of the shards"]
N2["valka-2<br/>owns about 1/3 of the shards"]
N3["valka-3<br/>owns about 1/3 of the shards"]
N1 <-->|"internal gRPC :50052"| N2
N2 <-->|"internal gRPC :50052"| N3
N3 <-->|"internal gRPC :50052"| N1
end
W1["Workers"] --> N1
W2["Workers"] --> N2
W3["Workers"] --> N3
N1 --> B[("Bucket<br/>leases · owner records<br/>WAL · snapshots")]
N2 --> B
N3 --> B
style N1 fill:#1F5B4F,stroke:#3E9F88,color:#fff
style N2 fill:#7c3aed,stroke:#8b5cf6,color:#fff
style N3 fill:#9E5A2E,stroke:#C27A48,color:#fff
style B fill:#336791,stroke:#2d5f8a,color:#fff
- Any node takes any request. It runs it locally, forwards it to the shard's owner, or asks every node and merges the answers.
- Workers attach to any node and run tasks owned by any node.
- A dead node's shards move to the others once its lease runs out (12–28 s in the failover benchmark). Nothing acknowledged is lost.
- Node-to-node calls never decide ownership. If they fail, requests are retried; the bucket still says who owns what.
How nodes share a bucket
All state is split into shards (default 4096, fixed when the bucket is first used). Each shard has exactly one owner at a time. The owner holds the shard in RAM and is the only node that writes its records. The rest of this section explains how that stays true when nodes crash, freeze or lose the network.
Leases: who is alive
Each node owns one object, nodes/{node}/lease. It holds the node's incarnation (bumped at
every start), its state (alive, draining or dead), a renew counter, a log watermark and
the node's addresses.
- Start. The node bumps its incarnation with a compare-and-swap (CAS). Two processes with the same node id can't both succeed; the older one is fenced.
- Renew. Every 2 s the node CASes its renew counter.
- Self-fence. A node that has not renewed for 7 s (lease timeout 10 s minus a 3 s safety
margin) stops serving and answers
UNAVAILABLEuntil it renews again. It goes quiet before anybody may take over its shards. - Watch others. Every 3 s each node lists
nodes/. A lease whose ETag has not changed for 10 s, measured on the observer's own clock, is suspect. No two clocks are ever compared. - Tombstone. The suspect's guardian (the first live node in rendezvous order for the
suspect's id) CASes the suspect's lease to
dead, conditioned on the exact ETag it watched. If the suspect renewed meanwhile, the CAS fails and nothing happens.
A network split between nodes never causes a takeover: both sides keep renewing their leases in the bucket. Leases are never deleted, because incarnation numbers must never be reused.
Owner records: who owns a shard
Each shard has an owner record, owners/{hh}/{shard}: the owning node, its incarnation, the
shard's generation and a state (owned or released). It is written only when the shard
moves, so a stable cluster never writes it.
A shard can be claimed only if its record is missing, released, or names an incarnation that
is dead and whose log is split. Every move is a CAS, so two nodes that briefly disagree
can't both win.
Placement: who should own a shard
Every node computes the same answer on its own with rendezvous hashing over the eligible
nodes: alive, not draining, past a 15 s join delay, and on a compatible protocol version. The
desired owner of shard s is the node with the highest hash(s, node). Adding or removing a
node moves only about 1/N of the shards. There is no leader.
Every 3 s each node runs a placement round:
- it claims shards whose desired owner is itself and that are claimable;
- it releases shards it owns whose desired owner is another node: at most 64 per round, and never within 5 minutes of the shard's last move, so a flapping node can't make shards bounce.
Operators can pin a shard to a node with the admin API. A pin wins while that node is eligible, and moves the shard at once.
On a cold start (no node past its join delay) every node takes part at once.
Claim and release
A claim of shard S by node B:
- Read
owners/Sand check it is claimable. - CAS it to
B, at generation + 1. - Load the newest snapshot of
S. - Write an acquisition snapshot at the new generation, pointing into
B's own log. - Serve
S.
Step 4 makes failover exact: the newest snapshot of a shard always points into its current owner's log, so the next claimer has exactly one log to replay. A claim that crashes between steps 2 and 4 is harmless; the next claimer takes the next generation from the same snapshot.
A release by the owner A (a planned move, a drain, a rolling restart):
- Stop accepting commands for
S. Callers are told to retry at the new owner. - Wait until every record of
Sis durable. - Write a final snapshot.
- CAS
owners/Storeleased. - Drop
Sfrom RAM.
Why a frozen node can't lose acknowledged work
After each WAL segment PUT, and before releasing its acknowledgements, a node re-reads its own lease with a conditional GET. A tombstone is a write to that lease. So either the check ran before the tombstone (and the segment was already in the bucket, where the split will find it), or the check ran after it (and fails, so the records are never acknowledged). A node that wakes up from a long pause finds itself fenced and exits. Nothing it acknowledged is lost; nothing it did not acknowledge is served.
Splitting a dead node's log
A dead node's log holds records of many shards. Rather than have every claimer read the whole log, the guardian reads it once and turns it into per-shard snapshots:
- Read the dead incarnation's watermark from its frozen lease.
- List its segments from the watermark. The end is the first missing position.
- Write an empty sealed segment at that end, so a late PUT from the dead node fails.
- Read the segments from the watermark up to the seal.
- For each shard the dead incarnation owned: newest snapshot plus this log's records, written as a split snapshot.
- Write the split marker
splits/{node}/{inc}. The shards are now claimable. - Delete the dead log.
A restarted node splits its own previous incarnation the same way, so restart and takeover are
one code path. If a split stalls, another node may redo it after 30 s (split_timeout_ms).
sequenceDiagram
participant X as Node X
participant G as Guardian
participant B as Bucket
participant N as New owners
X->>B: renew lease every 2 s
Note over X: kill -9
G->>B: list nodes/ every 3 s
Note over G: lease ETag of X unchanged for 10 s
G->>B: CAS lease of X to dead
G->>B: seal the log of X, read it
G->>B: PUT split snapshots
G->>B: PUT split marker
N->>B: CAS owner records to the next generation
N->>B: load snapshot, PUT acquisition snapshot
Note over N: serve the shards
Garbage collection
The desired owner of shard 0 runs a garbage collector every 5 minutes. It removes dead logs once their split marker exists, old snapshots (the newest two per shard are kept), and split markers nothing references any more. Leases are never collected.
Queue partitions
A queue is split into partitions (default 16). Partition p of queue Q lives on shard
hash(Q, p) mod num_shards, so a queue's tasks are spread over up to 16 shards and up to 16
nodes.
- A create with an idempotency key always lands on the key's partition, so deduplication is exact, also across a change of the partition count.
- A create without a key prefers a partition whose shard the receiving node owns (no hop).
- Order is FIFO per partition, priority first. A queue with one partition is strictly FIFO.
- A queue's peak throughput is roughly
partitions × per-shard throughput. Set the partition count to at least the number of nodes you want serving the queue.
Change a queue's partitions with PUT /api/v1/queues/{queue}/partitions (1 to 4096). Both
directions are online. Growing: new partitions start taking new tasks at once. Shrinking:
the removed partitions take no new tasks but are still served, by local and remote workers,
until their tasks are done. For the task retention window after a change (24 h by default),
keyed creates also check a key's old partition, so idempotency stays exact across it. GET /api/v1/queues lists every queue's partitions
and the shard, desired owner and draining flag of each.
Request routing
Every REST and gRPC call goes through the API layer of the node that received it:
| Request | Runs on |
|---|---|
| Create task | The receiving node picks the partition and mints the id; the shard's owner creates it |
| Get, cancel, delete, signal, runs, checkpoints of a task | The owner of the shard in the task id |
| List tasks, dead letters, workers; clear tasks; cluster stats; shard map | Every live node, merged |
The owner is found from a cache, refreshed from NOT_OWNER replies (which carry a hint), and as
a last resort from the owner record in the bucket. While a shard is moving, callers get
UNAVAILABLE (HTTP 503) and retry; the SDKs do this for you.
A merged answer that missed a node carries the header
X-Valka-Partial: true; missing=<node ids>. A complete answer carries X-Valka-Partial: false.
Workers across nodes
Each node keeps, per queue, the local workers waiting for a task. The owner of a task hands it out in two ways:
- Local: when a task becomes runnable and a worker on the owner node is waiting, it is offered at once, as soon as its record is durable.
- Remote: a node whose workers wait for a queue long-polls the owners of that queue's
partitions (
Fetch). The owner reserves a runnable task for that node, in RAM only, and hands it over. The fetching node's worker then dispatches it through the owner, which records the dispatch with the worker and the worker's node.
Details that keep this safe:
- A fetched task no local worker takes goes straight back. An unused reservation lapses after 30 s.
- A dispatch whose worker vanished before the assignment reached it is returned with a
RunReturnedrecord: back toPENDING, without spending an attempt. - Heartbeats are batched per owner. Results and checkpoints are forwarded to the owner and acknowledged end to end.
- Cancels and signals go to the node recorded on the run, then to the worker.
Tasks nobody is waiting for stay in the owner's pending index. Nothing is pushed blindly.
Reattach after a reconnect
A worker that loses its node keeps running its tasks and reconnects with backoff. Its hello lists the runs it is still executing, and it resends every result not yet acknowledged. For each running run, the owner answers:
- Reattached: the run is current. The owner records the worker's new node (
RunReattached) so results, cancels and signals flow again. - Adopted: the owner died right after dispatching and its dispatch record was lost; the task
is
PENDINGat exactly that attempt. The owner records the dispatch under the worker's run id instead of running the task a second time. - Stale: the run is over (cancelled, expired, finished elsewhere). The worker gets a
TaskCancellationand stops it.
After claiming shards, a node holds back recovered PENDING tasks for 5 s
(reconcile_grace_ms), so reconnecting workers can adopt their runs before anything is
dispatched again.
The SDKs reconnect to the address they were given. Point workers at a load balancer or a DNS name that covers several nodes, so a worker whose node died lands on a live one.
Configuration
A cluster needs:
- an S3-compatible bucket. The
localandmemorybackends only work inside one process; clustered mode refuses them; VALKA_CLUSTER__ENABLED=trueon every node;- the same
VALKA_CLUSTER__TOKENon every node. Clustered mode refuses to start without one; - a distinct, stable
VALKA_NODE_IDper node (1–64 characters fromA-Z a-z 0-9 _ -). The default is the host name; - the internal port (
:50052) reachable between nodes, and the advertised addresses set to something the other nodes can reach.
At start, a clustered node runs a storage self-test: create-if-absent must refuse to
overwrite, If-Match must reject a wrong ETag and accept the right one, and a new object must
be readable and listed at once. A bucket that fails is refused.
Environment variables
| Variable | Default | Description |
|---|---|---|
VALKA_NODE_ID | host name | Stable node id (distinct per node). A restart with the same id splits what the previous run left, so its shards are served again at once |
VALKA_CLUSTER__ENABLED | false | Allow other nodes on this bucket (S3 backend only) |
VALKA_CLUSTER__TOKEN | - | Shared secret every node-to-node call carries. Required when enabled |
VALKA_CLUSTER__NUM_SHARDS | 0 | Shard count of a new bucket: a power of two, 16 to 65536. 0 means the bucket's count, or 4096 |
VALKA_CLUSTER__INTERNAL_ADDR | 0.0.0.0:50052 | Node-to-node listener. Never expose it publicly |
VALKA_CLUSTER__ADVERTISE_INTERNAL_ADDR | internal addr | Address other nodes use to reach this node's internal listener |
VALKA_CLUSTER__ADVERTISE_GRPC_ADDR | VALKA_GRPC_ADDR | Public gRPC address shown for this node |
VALKA_CLUSTER__ADVERTISE_HTTP_ADDR | VALKA_HTTP_ADDR | Public REST address shown for this node |
VALKA_CLUSTER__TLS_CERT, VALKA_CLUSTER__TLS_KEY, VALKA_CLUSTER__TLS_CA | - | Optional mTLS for internal traffic (PEM files). Set all three to enable it |
VALKA_CLUSTER__LEASE_INTERVAL_MS | 2000 | Lease renewal period |
VALKA_CLUSTER__LEASE_TIMEOUT_MS | 10000 | An unchanged lease is declared dead after this |
VALKA_CLUSTER__LEASE_SAFETY_MARGIN_MS | 3000 | Stop serving at timeout − margin without a renewal |
VALKA_CLUSTER__MEMBERSHIP_INTERVAL_MS | 3000 | How often nodes/ is listed |
VALKA_CLUSTER__JOIN_DELAY_MS | 15000 | A new node takes shards after this |
VALKA_CLUSTER__PLACEMENT_INTERVAL_MS | 3000 | Placement round period |
VALKA_CLUSTER__CLAIM_CONCURRENCY | 64 | Shard claims in flight at once |
VALKA_CLUSTER__HANDOVER_PER_TICK | 64 | Shards released per placement round |
VALKA_CLUSTER__MIN_RESIDENCY_SECS | 300 | A shard stays put this long after a move |
VALKA_CLUSTER__SPLIT_TIMEOUT_MS | 30000 | Another node may redo a stalled split after this |
VALKA_CLUSTER__RECONCILE_GRACE_MS | 5000 | Hold recovered PENDING tasks for reattaching workers |
VALKA_CLUSTER__GC_INTERVAL_SECS | 300 | Garbage collector cadence |
VALKA_CLUSTER__STARTUP_LEASE_CHECK | true | Refuse to start while another live process holds this node id |
VALKA_ADMIN_TOKEN | - | Enables the admin API |
[cluster] in valka.toml
The same settings in the config file (from deploy/valka.example.toml):
[cluster]
# Allow other nodes on this bucket. Needs the s3 backend (local/memory CAS only works
# inside one process) and a token.
enabled = false
# Shared secret every node-to-node call carries. Required when enabled.
# token = ""
# Shard count of a NEW bucket (written to cluster.json on first start; fixed afterwards).
# Power of two, 16..=65536. 0 = whatever the bucket says, or 4096 for a new bucket.
num_shards = 0
# Node-to-node listener. Never expose it publicly.
internal_addr = "0.0.0.0:50052"
# Addresses other nodes / clients use to reach this node (default: the listen addresses).
# advertise_internal_addr = "10.0.0.1:50052"
# advertise_grpc_addr = "10.0.0.1:50051"
# advertise_http_addr = "10.0.0.1:8989"
# Optional mTLS for internal traffic (PEM files). The token is still required.
# tls_cert = "/etc/valka/tls/node.crt"
# tls_key = "/etc/valka/tls/node.key"
# tls_ca = "/etc/valka/tls/ca.crt"
# Lease renewal period, and how long an unchanged lease takes to be declared dead
# (by the observer's own clock). A node that has not renewed for
# lease_timeout - lease_safety_margin stops serving on its own.
lease_interval_ms = 2000
lease_timeout_ms = 10000
lease_safety_margin_ms = 3000
# How often nodes/ is listed (membership).
membership_interval_ms = 3000
# A new node takes shards after this (all nodes at once on a cold start).
join_delay_ms = 15000
# Placement round period, claims in flight, shards handed over per round.
placement_interval_ms = 3000
claim_concurrency = 64
handover_per_tick = 64
# A shard stays where it is for at least this long after a move.
min_residency_secs = 300
# Another node may redo a dead node's log split after this.
split_timeout_ms = 30000
# After claiming shards, recovered PENDING tasks wait this long so reconnecting workers
# can reattach their runs first.
reconcile_grace_ms = 5000
# Garbage collector cadence (dead logs, old snapshots, split markers).
gc_interval_secs = 300
# Refuse to start while another live process holds this node id.
startup_lease_check = trueThe node id and admin token are top-level keys (node_id, admin_token), not part of
[cluster]. Lease and placement timings are best left at their defaults: they set the failover
time (about 10–15 s) and keep false takeovers rare.
A three-node cluster with Docker Compose
The repository ships docker-compose.cluster.yml: three nodes on one S3-compatible store
(pgsty/silo, a MinIO fork).
git clone https://github.com/IWhitebird/Valka.git && cd Valka
docker compose -f docker-compose.cluster.yml up -d --build| Node | REST API and dashboard | gRPC |
|---|---|---|
| valka-1 | http://localhost:8981 | localhost:50061 |
| valka-2 | http://localhost:8982 | localhost:50062 |
| valka-3 | http://localhost:8983 | localhost:50063 |
Any node takes any request, and workers may attach to any node. The compose file sets a cluster
token and an admin token (change-me-cluster-token, change-me-admin-token): change both
before you use it anywhere but your laptop. The nodes reach each other on the compose network
at valka-N:50052; that port is not published.
Watching a takeover
Open the Cluster page at http://localhost:8981/cluster, or watch membership from the API:
watch -n1 "curl -s localhost:8981/api/v1/cluster/members | jq -c '.[] | {node_id, status, lease_age_ms}'"Then break a node:
# kill -9 a node
docker compose -f docker-compose.cluster.yml kill -s SIGKILL valka-2
# freeze a node (like a long GC pause): it is fenced when it wakes up
docker compose -f docker-compose.cluster.yml pause valka-3
docker compose -f docker-compose.cluster.yml unpause valka-3After a kill -9, the node's lease stops changing. About 10 s later it turns suspect, its
guardian tombstones it and splits its log, and the other nodes claim its shards.
GET /api/v1/cluster reports health.unowned_shards back at 0. Workers that were attached to
the dead node reconnect and reattach their running tasks.
Bring it back with docker compose -f docker-compose.cluster.yml start valka-2. It starts a new
incarnation and rejoins after the join delay. Shards that just moved stay where they are for
min_residency_secs (5 minutes), then placement hands the node its share again.
A frozen node is taken over the same way. When it wakes, its own-lease check fails: it exits and Docker restarts it as a new incarnation.
The metrics tell the same story: valka_tombstones_total, valka_splits_total,
valka_shard_claims_total, valka_reattach_total. To break nodes at random under load and
check that nothing acknowledged was lost, run the chaos test.
Admin API
The admin API changes where things live. It is disabled until you set VALKA_ADMIN_TOKEN
(or admin_token in valka.toml). Every admin request needs
Authorization: Bearer <admin token>; without a configured token the endpoints answer 403,
and with a wrong token 401. Every action is written to the bucket as an audit record.
| Method and path | Body | What it does |
|---|---|---|
POST /api/v1/cluster/nodes/{node}/drain | - | The node hands every shard over (final snapshots, nothing to replay), marks its lease dead and exits. 202 |
POST /api/v1/cluster/shards/{shard}/move | {"to": "node-id"} | Pins the shard to an eligible node. Its owner hands it over on its next placement round. 202 |
DELETE /api/v1/cluster/shards/{shard}/pin | - | Removes the pin: back to rendezvous placement |
PUT /api/v1/queues/{queue}/partitions | {"partitions": n} | Sets a queue's partition count (1 to 4096) |
GET /api/v1/cluster/audit?limit=50 | - | The newest admin actions (limit up to 500) |
Read-only views need no token: GET /api/v1/cluster/placement (pins, eligible nodes, effective
protocol) and GET /api/v1/queues (partition counts). See the REST API
for response shapes.
H="Authorization: Bearer $VALKA_ADMIN_TOKEN"
# Drain a node (decommission, maintenance)
curl -X POST -H "$H" localhost:8981/api/v1/cluster/nodes/valka-3/drain
# Pin shard 42 to valka-1, then return it to automatic placement
curl -X POST -H "$H" -H 'Content-Type: application/json' \
-d '{"to": "valka-1"}' localhost:8981/api/v1/cluster/shards/42/move
curl -X DELETE -H "$H" localhost:8981/api/v1/cluster/shards/42/pin
# Spread a hot queue over 32 partitions
curl -X PUT -H "$H" -H 'Content-Type: application/json' \
-d '{"partitions": 32}' localhost:8981/api/v1/queues/emails/partitions
# Who did what
curl -H "$H" 'localhost:8981/api/v1/cluster/audit?limit=20'A drained node exits cleanly. A supervisor that restarts on any exit (Docker's
restart: unless-stopped does) brings it back as a new incarnation, and it rejoins. Stop the
service for good when you decommission a node.
A pin to a node that is not eligible (dead, draining, or younger than the join delay) is ignored until that node becomes eligible.
Security
- Cluster token. Every internal call carries
cluster.token, compared in constant time. A wrong token shows asUnauthenticatedin the logs. - Optional mTLS. Set
tls_cert,tls_keyandtls_caon every node. Each node presents its certificate and requires one signed by the cluster CA. The token is still required. The certificates must be valid for the advertised internal host names. - Internal port.
:50052must be reachable between nodes and never from the public internet. - Admin token. Keep
VALKA_ADMIN_TOKENseparate from the cluster token, and unset it where you don't need the admin API. - Public API. The REST and gRPC APIs on
:8989and:50051have no authentication of their own (apart from the admin endpoints). Keep them on a private network or behind a proxy that authenticates.
Peer clients have a small circuit breaker: after 3 consecutive transport failures to a peer,
calls to it fail fast for 3 s, then one call is let through to probe it. This only shortens
UNAVAILABLE answers during an outage; it never changes ownership.
Failure handling
| What happens | What the cluster does | What you see |
|---|---|---|
kill -9 on a node | After 10 s its guardian tombstones it, splits its log (about 1 s), and the desired owners claim its shards | Its shards are unavailable for about 11–15 s. Its workers reconnect and reattach. Nothing acknowledged is lost |
| Node frozen, then wakes up | Taken over while frozen. On waking, its PUTs hit the seal, its own-lease check fails: it exits and restarts as a new incarnation | Requests it held fail; clients retry |
| Node cut off from the bucket | It acknowledges nothing. After 7 s it stops serving; after 10 s it is tombstoned and its shards move | About 15 s of unavailability for its shards |
| Network split between nodes, bucket reachable | No takeover: leases keep renewing. Forwarding across the split fails | Cross-side requests get UNAVAILABLE until it heals |
| Bucket down for everyone | No acks anywhere; every node stops serving; nobody can tombstone anybody | Full outage, recovers by itself |
| Two nodes claim one shard | CAS on the owner record: one wins | Nothing |
| Crash in the middle of a split | Another node redoes it after split_timeout_ms | Takeover delayed by up to 30 s |
| Same node id started twice | The lease CAS at start lets one win; the other exits | Nothing (fix the duplicate id) |
| Node joins | After the join delay, about 1/N of the shards move by release and claim, rate-limited | Brief retried UNAVAILABLE per moved shard |
SIGTERM (rolling restart) | With live peers: release every shard with a final snapshot, then exit. Alone: keep the shards and reclaim them on restart | Near-zero unavailability |
| A worker's node dies mid-task | The worker keeps running, reconnects, reattaches; its result is resent and acknowledged | The task completes once |
| The owner dies right after dispatching | The worker's reattach adopts the run | No duplicate run |
Limits
- Useful nodes are about
num_shards / 8. Pick the shard count of a new bucket with that in mind; it can never change. - A queue partition is served by one shard (one owner, one lock). Add partitions for a hot queue.
- Every node relays every other node's events. That fan-in is fine up to a few dozen nodes.
- Per node, coordination costs about one lease PUT every 2 s, one own-lease GET per WAL flush and
one
nodes/list every 3 s. None of it grows with the number of nodes.
For alerts and what to do when they fire, see Deployment.