Architecture
How Valka works under the hood, from one node to many.
The one rule
The bucket is the only truth. Valka keeps nothing important on a node's disk. It uses no database, no Raft, no ZooKeeper and no message broker. Everything that matters is an object in one S3-compatible bucket, or in a local directory for a single dev node:
- every change to a task is a record in a write-ahead log (WAL) stored as objects;
- the live state is held in RAM by the node that owns it, and rebuilt from the bucket whenever ownership moves;
- who owns what, and who is alive, is decided by conditional writes (compare-and-swap on an object's ETag) in the same bucket.
A node is therefore disposable. Kill it, and another node rebuilds its state from the bucket. The price is latency: an acknowledgement waits for an S3 PUT, typically 50–200 ms with the default 50 ms group-commit window. In return you get durability, simple operations and a cluster that needs nothing but a bucket.
The big picture
graph TD
P["Producers<br/>REST · gRPC · CLI · SDK clients"] --> A1
P --> A2
W1["Workers<br/>one gRPC stream each"] --> D1
W2["Workers<br/>one gRPC stream each"] --> D2
subgraph NA["Node A"]
A1[API layer] --> E1["Engine<br/>shards A owns, in RAM"]
D1[Dispatcher] --- M1[Matching]
E1 --> M1
E1 --> WR1[WAL writer]
end
subgraph NB["Node B"]
A2[API layer] --> E2["Engine<br/>shards B owns, in RAM"]
D2[Dispatcher] --- M2[Matching]
E2 --> M2
E2 --> WR2[WAL writer]
end
A1 <-->|"internal gRPC :50052<br/>token, optional mTLS"| A2
WR1 --> B[("Bucket<br/>leases · owner records · WAL · snapshots")]
WR2 --> B
style E1 fill:#1F5B4F,stroke:#3E9F88,color:#fff
style E2 fill:#1F5B4F,stroke:#3E9F88,color:#fff
style B fill:#9E5A2E,stroke:#C27A48,color:#fff
style D1 fill:#7c3aed,stroke:#8b5cf6,color:#fff
style D2 fill:#7c3aed,stroke:#8b5cf6,color:#fff
Things to notice:
- Any node accepts any request. A request for a task another node owns is forwarded to
that node over the internal gRPC listener (default
:50052). - Workers can attach to any node and run tasks owned by any node.
- Nodes never ask each other who is alive or who owns what. They read it from the bucket. Node-to-node calls only move requests and tasks around. If those calls fail, requests are retried, but ownership does not change.
- One node is a cluster of one. A single
valka-serverruns exactly the same protocol, alone. There is no separate single-node code path.
What is in the bucket
| Key | Holds |
|---|---|
cluster.json | Cluster id, shard count and format version. Written once |
nodes/{node}/lease | Who is alive: incarnation, state, renew counter, log watermark, addresses |
owners/{hh}/{shard} | Which node incarnation owns a shard, and the shard's generation |
queues/{hh}/{queue} | A queue's partition count |
wal/{node}/{inc}-{seq}.seg | Each node's own log, append-only |
snapshots/{hh}/{shard}/... | Per-shard state checkpoints |
splits/{node}/{inc} | Marker: this dead log has been redistributed |
logs/{hh}/{run}/... | Task log lines (not part of the WAL) |
placement.json, admin/... | Operator shard pins, admin audit records |
{hh} is two hex digits of a hash of the rest of the key. It spreads busy key families over
the store's partitions. WAL segments are zstd-compressed, CRC-framed JSON, so anything can
read the log straight from the bucket.
Shards, partitions and ids
Shards. All state is split into a fixed number of shards. The default is 4096; the count
is chosen when a bucket is first used, stored in cluster.json, and never changes. A shard is
the unit of ownership: at any moment exactly one node serves it, holds its state in RAM and
writes its records. A shard holds its tasks, runs, signals, dead letters and idempotency keys.
Nothing spans two shards, so no operation needs coordination between shards.
Queue partitions. A queue is split into partitions (default 16, changed with an admin
call). Partition p of queue Q lives on shard hash(Q, p) mod num_shards. A queue's tasks
are therefore spread over up to 16 shards, which may be owned by up to 16 different nodes.
When a task is created, the receiving node picks a partition:
- with an idempotency key:
jump_hash(key, partitions), so the same key always lands on the same shard and deduplication is exact; - without a key: preferably a partition whose shard this node owns (no network hop), otherwise a random one.
Ids carry their shard. Task ids are UUIDv7 (time-ordered) with the shard number embedded in the low 16 bits. Any node can route a get, cancel or signal straight to the right shard without a lookup. Run ids are minted in the same shard as their task.
Inside one node
A valka-server process runs these pieces in one tokio runtime:
| Piece | Crate | What it does |
|---|---|---|
REST API (:8989) and gRPC API (:50051) | valka-server | Public endpoints. The dashboard is served from the REST port |
| API layer | valka-server | Decides where a request runs: locally, forwarded to the owner, or scattered to every node and merged |
Internal listener (:50052) | valka-server | Answers other nodes' Execute, Scatter and Events calls |
| Engine | valka-engine | The shards this node owns, the command API, timers, the pending index, snapshots, recovery and the control loop |
| WAL writer | valka-wal | Batches records into segments and PUTs them (group commit) |
| Lease keeper | valka-wal | Renews this node's lease every 2 s |
| Dispatcher | valka-dispatcher | One bidirectional gRPC stream per worker: assignments, results, heartbeats, signals, cancels |
| Matching | valka-matching | Which local workers wait for which queue. Hands a task to one of them |
| Remote feeder | valka-server | When local workers wait for a queue whose partitions other nodes own, long-polls those owners for work |
| Event relay | valka-server | Streams other nodes' task events to this node's subscribers (SSE, gRPC) |
| Log ingester | valka-engine | Batches workers' log lines into logs/ objects |
The engine runs four background loops:
- the ticker, every 100 ms: lease expiry, retry promotion, delayed tasks, retention of finished tasks, coalesced heartbeat records;
- the feeder: offers runnable tasks to waiting workers;
- the snapshotter: writes shard snapshots and truncates the log;
- the control loop: membership, placement, guardian duties, garbage collection.
There is no scheduler leader. Each node runs the timers of the shards it owns.
Crates
| Crate | Purpose |
|---|---|
valka-proto | Generated gRPC stubs from the proto files |
valka-core | Shared types, configuration, errors, metrics, shard and id helpers |
valka-wal | Object-store WAL: store wrapper, segment codec, group-commit writer, snapshots, leases, owner records, splits |
valka-engine | In-RAM shard state machines, the command API, timers, pending index, snapshots, recovery, ownership |
valka-matching | In-memory matching of tasks to waiting workers |
valka-dispatcher | Worker stream management, heartbeats, task dispatch, signal delivery, reattach |
valka-cluster | Internal RPC commands, peer clients, cluster token and mTLS |
valka-server | The server binary: REST, gRPC, internal listener, routing, admin API |
valka-sdk | Rust worker and client SDK |
valka-cli | Command-line interface |
valka-tests | Unit and integration tests |
The write path: acked means durable
Every change goes through one function, Engine::mutate: a create, a dispatch, a result, a
cancel, a signal or a timer firing.
sequenceDiagram
participant C as Caller
participant E as Engine
participant W as WAL writer
participant B as Bucket
C->>E: command
E->>E: lock shard, validate against RAM
E->>E: build record, apply it to RAM
E->>W: append record
E->>E: unlock shard
Note over W: buffer up to 50 ms or 4 MB
W->>B: PUT segment
B-->>W: 200
W->>B: conditional GET of own lease
B-->>W: still mine
W-->>E: durable, in order
E->>E: publish events, offer to matching
E-->>C: acknowledge
- Apply, then wait. RAM is updated under the shard lock before the record is durable, so a shard's records are produced in order and RAM is always at or ahead of the durable log. If the PUT never succeeds, the caller is told it failed and the node exits. Its supervisor restarts it, and the bucket is the truth again.
- Group commit. One segment carries every record appended during the window, from all shards. Throughput is bounded by PUT bandwidth, not PUT latency. Up to 4 segments are in flight at once; acknowledgements are still released strictly in order.
- The own-lease check. After each segment PUT, before releasing its acks, the writer does a conditional GET of its own lease. If another node has declared this one dead, or a newer process took the same node id, the check fails, the acks fail, and the node exits. That one request per flush is the whole write fence. It needs no clock, and its cost does not grow with the cluster.
- Only durable work is visible to workers. A task is offered to matching only after its
TaskCreatedrecord is durable, so a worker never runs a task that could vanish. - Replay is deterministic. A record whose precondition no longer holds is a no-op.
Timers that fire produce their own records (
LeaseExpired,TaskPromoted), so rebuilding a shard never consults the clock.
Life of a task
On one node
sequenceDiagram
participant P as Producer
participant N as Node
participant W as Worker
P->>N: CreateTask
N->>N: pick partition, mint id
N->>N: TaskCreated, durable
N-->>P: task (PENDING)
N->>W: TaskAssignment (a worker was waiting)
N->>N: TaskDispatched (async)
W->>N: Heartbeat (extends the run lease)
W->>N: Checkpoint (optional, acked once durable)
W->>N: TaskResult
N->>N: RunCompleted, durable
N-->>W: ResultAck APPLIED
- If no worker is waiting, the task stays in the shard's pending index in RAM (by priority, then age). The feeder offers it as soon as a worker for that queue is free. There is no polling and no database query on this path.
- A run holds a lease, extended by worker heartbeats. If the worker vanishes, the timer
fires
LeaseExpiredand the task is retried or dead-lettered. - Results are acknowledged end to end:
APPLIED,STALEorRETRY. SDKs keep a result until it isAPPLIEDorSTALE, so a result produced while the server restarts is not lost.
See Task Lifecycle for states, retries, signals and checkpoints.
Across nodes
With three nodes, the producer may talk to node A, the task's shard may be owned by node B, and the only free worker may be attached to node C:
sequenceDiagram
participant P as Producer
participant A as Node A
participant B as Owner B
participant C as Node C
participant W as Worker on C
P->>A: CreateTask
A->>A: pick partition, owner is B
A->>B: Execute CreateTask
B->>B: TaskCreated, durable
B-->>A: task
A-->>P: task (PENDING)
C->>B: Fetch emails (long poll)
B-->>C: task, reserved for C in RAM
C->>B: Dispatch run to worker on C
B->>B: TaskDispatched (worker, node C)
C->>W: TaskAssignment
W->>C: TaskResult
C->>B: result
B->>B: RunCompleted, durable
B-->>C: APPLIED
C-->>W: ResultAck APPLIED
- A task C fetched but could not hand to a worker goes back. An unused reservation lapses after 30 s.
- A run B recorded that never reached its worker goes back to
PENDINGwithRunReturned, without spending an attempt. - The owner records which node each run's worker is on, so cancels and signals reach it.
- Heartbeats are batched per owner. Checkpoints and results are forwarded to the owner and acknowledged once durable.
- Event subscribers on any node see every node's events through the event relay.
Worker streams
Each worker keeps one bidirectional gRPC stream with one node. Over it:
- Server to worker: task assignments, cancellations, signals, heartbeat acks, result acks,
and
ServerShutdownwhen the node stops. - Worker to server: the hello (name, queues, concurrency, and the runs it is still
executing), results, heartbeats, log batches, signal acks, and
GracefulShutdown.
sequenceDiagram
participant W as Worker
participant S as Node
W->>S: Session (open stream)
W->>S: WorkerHello (queues, running runs)
S->>W: TaskAssignment
W->>S: Heartbeat (every 10 s)
S->>W: HeartbeatAck
W->>S: LogBatch
S->>W: TaskSignal
W->>S: SignalAck
W->>S: TaskResult
S->>W: ResultAck
Step checkpoints use a separate unary call, WorkerService.Checkpoint, so a handler can wait
until its step is durable. See gRPC API.
Snapshots, truncation and restart
The WAL would grow forever unless it is cut. The snapshotter writes each changed shard's whole state to the bucket every 60 s, after 50,000 records, or once 64 MiB of log has accumulated (the log budget), whichever comes first.
- A snapshot covers only durable records. The shard is captured, then the writer is synced. If that sync fails, nothing is written.
- Replay skips records at or below the snapshot's sequence number and records of other generations, so it never depends on records being idempotent.
- Truncation: once every shard a node serves has a snapshot pointing past a segment, the segment is deleted. The node first raises a watermark in its lease, then deletes in ascending order.
- The newest two snapshots per shard are kept.
Restart of a node with the same VALKA_NODE_ID: the new process takes its lease with a
new incarnation number, which fences the previous process if it is somehow still running.
It then splits its previous incarnation's log, which makes its shards claimable at once,
with no lease timeout to wait out. Alone, it claims them right back. In a cluster, live peers
claim them immediately, and placement moves its share back once it has been up for the join
delay (15 s) and the shards have stayed put for the minimum residency (5 min). Restart cost is
bounded by the log budget: at most 64 MiB of log to read.
Many nodes
Several nodes share one bucket without any coordination service:
- Leases say who is alive. Each node renews
nodes/{node}/leaseevery 2 s. A lease that stops changing for 10 s is declared dead by another node, its guardian. - Owner records say who owns each shard. They change only by compare-and-swap, so two nodes can never both win a shard.
- Placement uses rendezvous hashing over the live nodes. Every node computes the same answer; there is no leader.
- A dead node's log is split once into per-shard snapshots, then its shards are claimed by their new owners.
The full protocol, configuration and operations are on the Clustering page.
Failure handling
| What happens | What the cluster does | What you see |
|---|---|---|
kill -9 on a node | After 10 s its guardian tombstones it, splits its log (about 1 s), and the new owners claim its shards | Its shards are unavailable until claimed: 12–28 s in the failover benchmark, where claiming dominates on a slow bucket (UNAVAILABLE, clients retry). Its workers reconnect and reattach. Nothing acknowledged is lost |
| Node frozen 30 s, then wakes up | Taken over while frozen. On waking, its own-lease check fails: it exits and restarts as a new incarnation | Requests it held fail; clients retry |
| Node cut off from the bucket | It acknowledges nothing. After 7 s it stops serving; after 10 s it is tombstoned and its shards move | About 15 s of unavailability for its shards |
| Network split between nodes, bucket reachable | No takeover: leases keep renewing. Forwarding across the split fails | Cross-side requests get UNAVAILABLE until it heals |
| Bucket down for everyone | No acks anywhere; every node stops serving; nobody can tombstone anybody | Full outage, recovers by itself |
| Node restarts with the same id | The new incarnation fences the old one and splits its own old log at once; peers claim its shards (alone: it claims them itself); its share moves back later | Seconds of unavailability for those shards |
SIGTERM | With live peers: release every shard with a final snapshot, then exit. Alone: snapshot and keep the shards | Near-zero unavailability |
| A worker's node dies mid-task | The worker keeps running, reconnects, reattaches; its result is resent | The task completes once |
| A worker dies mid-task | The run's lease expires; the task is retried | One retry (checkpointed steps are skipped) |
Guarantees
Guaranteed under every failure above:
- Acknowledged means durable. A create, result, cancel, checkpoint or signal that got a success response is in the bucket and survives any node failure.
- One history per shard. Two nodes never write diverging histories of the same shard.
- Bounded recovery. At most 64 MiB of log per node to replay or split.
- Failover. A dead node's shards are served again within the lease timeout plus the claims (12–28 s measured), as long as some node can reach the bucket.
- Exact idempotency. Two creates with the same idempotency key in the same queue produce one task.
Not guaranteed, by design:
- Exactly-once execution. Execution is at-least-once: a task can run twice around failures. Make handlers idempotent, or use steps.
- Global ordering. FIFO holds per queue partition, priority first. Set a queue's partitions to 1 for strict FIFO.
- Durable reads. Reads are served from RAM and can show a write whose acknowledgement is still in flight.
- Low-latency acks. An acknowledgement waits for an object-store PUT.
Event broadcasting
Every state change emits a task event on a tokio::sync::broadcast channel. Each node also
relays the events of every other live node, so a subscriber on any node sees the whole
cluster. Consumers:
- gRPC event streams:
ApiService.SubscribeEvents - SSE endpoint:
GET /api/v1/events - Web dashboard: uses SSE for live updates