Introduction
What is Valka and why does it exist?
What is Valka?
Valka is a distributed task queue built entirely in Rust with a single external dependency: an S3-compatible bucket. For a single node, even that can be a local directory.
Most task queues bolt together a message broker, a database, and a cache. Every moving part is another thing to deploy, monitor, and debug at 3 AM. Valka takes a different approach: the bucket is the single source of truth. Every state change is a record in a write-ahead log stored in the bucket, and nothing is acknowledged before its record is there. An in-memory matching engine and gRPC bidirectional streaming replace the message broker entirely.
The result: nodes you can kill at any time, nothing to operate but a bucket, and a fast path that never touches a database.
Key Features
- One dependency: if you have a bucket (AWS S3, MinIO or another S3-compatible store), you can run Valka. A single node can also use a local directory
- Acked means durable: a task, result or cancel is acknowledged only once it is in the bucket
- Disposable nodes: live state is in RAM and rebuilt from the bucket, so any node can be killed or replaced
- One node or many: point more nodes at the same bucket. Any node takes any request, and a dead node's work moves to the others once its lease runs out (12–28 s, see Benchmarks)
- Workers ride through node failures: they reconnect and reattach their running tasks
- In-memory hot path: a task reaches a waiting worker the moment it is durable, no polling
- Polyglot SDKs: Rust, TypeScript, Go, and Python, with the same worker model
- gRPC bidirectional streaming: single connection per worker, real-time task delivery
- Automatic retries: exponential backoff with dead letter queue
- Step checkpoints: a retried task skips the steps an earlier attempt finished
- Task signals: send real-time signals to running workers
- Observable: real-time log streaming, event feeds, Prometheus metrics, web dashboard
- Idempotency keys: prevent duplicate tasks
- Scheduled tasks: delayed and deferred task execution
How It Works
graph TD
C[REST / gRPC Client] --> API["API layer<br/>any node"]
API -->|routes to the shard owner| E["Engine<br/>shards in RAM"]
E -->|group commit| B[("Bucket<br/>write-ahead log + snapshots")]
E -->|once durable| MS["Matching<br/>waiting workers"]
E -.->|no worker waiting| PI["Pending index<br/>in RAM"]
PI -.->|worker frees up| MS
MS --> D[Dispatcher<br/>gRPC Bidi Stream]
D --> W1[Worker A]
D --> W2[Worker B]
D --> W3[Worker C]
style E fill:#1F5B4F,stroke:#3E9F88,color:#fff
style MS fill:#059669,stroke:#10b981,color:#fff
style B fill:#9E5A2E,stroke:#C27A48,color:#fff
style D fill:#7c3aed,stroke:#8b5cf6,color:#fff
Write path: a new task is applied to the owning node's RAM state and appended to the write-ahead log. Records from many requests are batched into one segment (group commit) and written to the bucket. Only then is the task acknowledged.
Hot path: if a worker is already waiting, the durable task is handed to it in memory and pushed over its gRPC stream at once.
Waiting tasks: if no worker is waiting, the task stays in the owner's pending index in RAM. It is offered as soon as a worker for its queue frees up. No polling, no database query.
SDKs
| Language | Package | Install |
|---|---|---|
| Rust | valka-sdk | cargo add valka-sdk --git https://github.com/IWhitebird/Valka |
| TypeScript | @valka/sdk | npm install https://github.com/IWhitebird/Valka/releases/download/v0.2.0/valka-sdk-0.2.0.tgz |
| Go | github.com/IWhitebird/Valka/sdks/go | go get github.com/IWhitebird/Valka/sdks/go |
| Python | valka | pip install "git+https://github.com/IWhitebird/Valka@v0.2.0#subdirectory=sdks/python" |
Any language can also interact via the REST API on port 8989.
Next Steps
- Quick Start: get Valka running in minutes
- Architecture: deep dive into how Valka works
- Task Lifecycle: understand task states and transitions
- Clustering: run many nodes on one bucket