valka

Benchmarks

Throughput, latency and failover measured on one laptop, how to reproduce them, and what they found.

Measured on one laptop, so read the numbers as an order of magnitude and as a comparison between setups, not as what a production cluster on S3 does. Every run is reproducible with the commands below.

Setup

ItemValue
MachineIntel Core i5-12450H (8 cores, 12 threads), 15 GB RAM, Linux 6.14, NVMe
Valka0.2.0, release build, default configuration (50 ms group commit, 4096 shards)
CPU capssingle node: 3 CPUs (systemd-run -p CPUQuota=300%); three nodes: 1.5 CPUs each; MinIO: 1.5 CPUs; load generator: 3 CPUs
Object storelocal disk (local backend), or MinIO on the same machine (pgsty/silo)
Loadexamples/rs/bench.rs: N creates in flight for 30 s, 100-byte inputs, one worker per node with 256 slots and a handler that returns at once

The load generator reports the create ack (request sent → task durable and acknowledged) and the pickup (create sent → the worker's handler starts), and checks that every acknowledged task ran.

Results

One node, local disk

In flightCreates/sAck p50Ack p99Pickup p50Pickup p99Ran
641,22552.2 ms53.8 ms52.9 ms54.8 msall 36,800
2564,66154.8 ms61.1 ms59.6 ms81.5 msall 140,032
1,02417,83056.9 ms75.1 ms80.8 ms307 msall 535,955

A clean stop after each run (final snapshots of every dirty shard) took 0.3 s, 0.8 s and 2.9 s. The server held 536k tasks in 1.7 GB of RAM at the end of the largest run.

Three nodes, MinIO

Producers spread over the nodes (each create may be forwarded to its shard's owner), one worker per node; a task is run by whichever node's worker takes it first.

In flightCreates/sAck p50Ack p99Pickup p50Pickup p99Ran
641,14655.3 ms99.2 ms54.2 ms79.3 msall 34,454
2564,52855.1 ms103.0 ms54.9 ms105.9 msall 136,223
1,02415,61363.0 ms131.5 ms63.5 ms131.9 msall 469,656

About two thirds of the creates reach a node that does not own their shard and are forwarded, which adds one hop and the owner's commit window to their acknowledgement.

Failover (three nodes in Docker, MinIO)

scripts/bench-failover.sh kills a node with kill -9 and polls another node for when the dead node is detected (its lease tombstoned) and when the cluster has recovered (every shard served by a live node again). Six trials per setup, a different node each time: once as docker-compose.cluster.yml ships (no CPU limits), once with every node and MinIO limited to 1.5 CPUs.

Six trialsDetectedRecovered (no limits)Recovered (1.5 CPUs each)
fastest9.4 s11.8 s17.9 s
median13.4 s15.0 s24.6 s
slowest15.4 s17.4 s27.8 s

Detection is the lease timeout (10 s) plus up to one membership listing (3 s). Recovery then takes one placement round and the claims themselves: about five object-store requests per shard, for ~1,400 shards per surviving node, so it is bounded by how many requests the bucket serves at once — here a single MinIO process. Workers keep running their tasks meanwhile and reattach them; creates and results for the dead node's shards get UNAVAILABLE and the SDKs retry them.

A five-minute chaos run on the same cluster (11 random kill -9s, freezes and SIGTERM restarts under load, scripts/chaos.sh) acknowledged 5,476 tasks and completed all of them, with none lost and no idempotency key used twice; 278 ran more than once, as at-least-once delivery allows around failures.

What bounds what

  • Ack latency is the group-commit window (50 ms) plus one PUT and one conditional GET of the node's lease. VALKA_WAL__FLUSH_INTERVAL_MS=10 gave 12.4 ms p50 and 14.1 ms p99 acks on one node at 64 in flight (5,168 creates/s, pickup p99 15.7 ms), at four times the PUT count (2,423 segments in 30 s instead of 575).
  • Throughput is bounded by CPU (JSON encoding and compression of segments, gRPC), not by PUT latency: up to four segments are in flight per node and each holds every record of its window.
  • Pickup is the ack latency plus about a millisecond while workers keep up: a task is handed to a waiting worker the moment it is durable.

Object-store requests and cost

A busy node writes at most one segment per commit window, 20 per second at 50 ms, plus one conditional GET of its lease per segment, one lease PUT every 2 s, one LIST of nodes/ every 3 s, and snapshots of dirty shards every minute. An idle node writes only its lease. In the single-node runs above the node wrote 525–572 segments in 30 s at every load: segments are cut by time, not size, so the PUT count does not grow with throughput.

At the S3 Standard list price of $0.005 per 1,000 PUT requests, a node that is busy around the clock writes about 1.7 million segments a day, about $8.60 a day; raising the window to 200 ms cuts that by four. The dashboard's storage page shows the estimate for your own cluster.

What the benchmarks found

The first runs found real bugs, each fixed with a regression test before these numbers were taken:

SymptomCause
A node under load stopped writing; even timers stoppedThe feeder handed back a refused task, which woke it at once: a busy loop that never yielded
About 20 tasks/s per worker, whatever its concurrencyThe server applied a worker's results one after another, each waiting for its group commit
Some tasks started 30 s lateA refused feeder batch dropped its other tasks until their offers lapsed
Workers fell behind as the backlog grewEvery task leaving PENDING scanned the whole pending index
Half of the tasks fetched from other nodes went back againDispatches to other owners ran one after another, with no slot open meanwhile
Recovery after kill -9 took up to 37 sClaims refused before the dead log was split backed off for up to 30 s
A clean stop with thousands of dirty shards took minutesSnapshots were written one after another

Reproducing

cargo build --release -p valka-server -p valka-examples --example bench

# One node on a local directory
VALKA_STORAGE__PATH=/tmp/valka-bench ./target/release/valka-server &
VALKA_BENCH_INFLIGHT=256 VALKA_BENCH_WORKER_CONCURRENCY=256 ./target/release/examples/bench

# Three nodes
docker compose -f docker-compose.cluster.yml up -d --build
VALKA_API_KEY=change-me-api-key \
VALKA_BENCH_GRPC=http://localhost:50061,http://localhost:50062,http://localhost:50063 \
  ./target/release/examples/bench

# Failover
scripts/bench-failover.sh

VALKA_BENCH_SECONDS, VALKA_BENCH_INFLIGHT, VALKA_BENCH_WORKER_CONCURRENCY and VALKA_BENCH_PAYLOAD change the load (see the top of examples/rs/bench.rs).

On this page