Open source · Apache-2.0 · written in Rust
A task queue whose only dependency is a bucket
Every task, retry and result is in your S3-compatible bucket before Valka says OK. Nodes hold their shards in memory and are disposable: kill one, and the others take its work over once its lease runs out.
$ export COMPOSE_FILE=docker-compose.cluster.yml$ docker compose up -d$ export VALKA_HTTP=http://localhost:8981$ export VALKA_API_KEY=change-me-api-key$ docker compose kill -s SIGKILL valka-2$ valka cluster statusCluster 01a1605b… health OK shards 4096 clusteredNODE INC STATUS SHARDS PENDING RUNNINGvalka-1 7 alive 1409 1 0valka-3 8 alive 2687 2 0valka-2 7 dead - - -
Saved before it is acknowledged
Every create, result and cancel is a record in a write-ahead log stored in your bucket. A node answers OK only after the segment holding it is written and the node has re-read its own lease, so a node that was declared dead cannot acknowledge anything.
One writer per shard
State is split into shards. Each has exactly one owner, recorded in the bucket and moved only by compare-and-swap. No database, no Raft, no gossip, no leader: nodes that share nothing but a bucket still never write the same shard.
Work survives the failure
Workers keep running when their node dies, reconnect anywhere and reattach their tasks. Results are acknowledged end to end, and every create carries an idempotency key, so retries never make a second task.
What happens to a write
The same path for a create, a result, a cancel or a retry timer firing.
Your request
REST or gRPC, to any node
Shard owner
validates and applies it in memory
WAL segment
group commit to the bucket, ≤ 50 ms
Lease check
still the owner? only then OK
A worker
gets the task the moment it is saved
Everything lives in your bucket
Plain objects you can list, read and back up. The log is compressed JSON, so any tool can follow it. Run one node against a local directory while you build, and a cluster against S3, MinIO or another compatible store in production.
How nodes share itcluster.json cluster id, shard count
nodes/valka-2/lease liveness: incarnation, renewals
owners/b1/02731 owner and generation of shard 2731
wal/valka-2/…07-…0120.seg one group commit of records
snapshots/b1/02731/…-split.snap shard 2731, rebuilt from a dead log
splits/valka-2/7 incarnation 7 is dead and split
queues/3f/emails partition count of a queue
logs/…/{run}/… task log linesWorkers in four languages
A worker lists a few nodes, takes tasks over one gRPC stream and survives any of them dying. Producers create tasks with the client of the same SDK, or with plain HTTP.
Every SDK reads VALKA_API_KEY when the cluster requires keys.
let worker = ValkaWorker::builder()
.name("email-worker")
.server_addrs(&["http://valka-1:50051", "http://valka-2:50051"])
.queues(&["emails"])
.concurrency(8)
.handler(|ctx| async move {
let input: Email = ctx.input().map_err(|e| e.to_string())?;
send(&input).await;
Ok(json!({ "sent": true }))
})
.build()
.await?;
worker.run().await?;Built for real work
- Steps
- Checkpoint each step; a retry skips the finished ones.
- Signals
- Send messages to a running task and get acknowledgements.
- Retries and dead letters
- Exponential backoff, then a dead-letter queue.
- Scheduled tasks
- Run no earlier than a given time.
- Idempotency keys
- One task per key and queue, on whichever node you ask.
- Queue partitions
- Spread a hot queue over nodes; resize online.
- Dashboard and CLI
- Tasks, workers, shard map, drains and shard moves.
- Metrics and runbooks
- Prometheus metrics, alert rules, what to do when they fire.
- API keys
- Keys for producers and workers, a separate admin token.
Tested the way it fails
Each bug below was found before release and fixed with a regression test.
7
protocol bugs found by the TLA+ model
before any cluster code existed; the model runs in CI
4
found by a randomized simulation
thousands of seeds of kills, freezes and bucket faults, with a history checker
3
found by chaos on real processes
kill -9, freezes and restarts under load on a three-node cluster
7
found by load and failover benchmarks
a livelock, 30-second stalls, a per-worker ceiling, slow takeovers
Try it in a minute
One node against a local directory, or three nodes and an S3-compatible store in Docker. The dashboard runs on every node.
# one node, data in ./data cargo run -p valka-server # three nodes: dashboard on :8981, :8982, :8983 docker compose -f docker-compose.cluster.yml up -d # break it and watch it recover scripts/chaos.sh