REST API
Complete REST API reference for task management, signals, monitoring, the cluster and the admin API.
The Valka REST API runs on port 8989 by default. All endpoints return JSON.
In a cluster, every node serves the whole API. A request about one task is forwarded to the node that owns the task's shard; a list is gathered from every live node and merged. You can send any request to any node.
Authentication
When the server is started with VALKA_API_KEYS (comma-separated), every /api/v1 endpoint
needs one of those keys:
Authorization: Bearer <api key>A missing or wrong key gets 401 with code UNAUTHENTICATED. The event stream also accepts the
key as ?access_token=<key>, because a browser's EventSource cannot send headers.
/healthz and /metrics stay open for probes and scrapers. The admin endpoints use the
separate admin token instead. Without VALKA_API_KEYS the API is open, and the server logs a
warning at startup.
Partial answers
Endpoints that gather from every node (GET /api/v1/tasks, DELETE /api/v1/tasks,
GET /api/v1/workers, GET /api/v1/dead-letters) set the X-Valka-Partial response header:
X-Valka-Partial: false
X-Valka-Partial: true; missing=valka-2true means some live node did not answer in time, and the list is missing its part. Retry, or
check the cluster.
Tasks
Create a Task
POST /api/v1/tasks{
"queue_name": "emails",
"task_name": "send-welcome-email",
"input": { "to": "user@example.com", "subject": "Welcome!" },
"priority": 0,
"max_retries": 3,
"timeout_seconds": 300,
"idempotency_key": "welcome-user-123",
"metadata": { "source": "signup-flow" },
"scheduled_at": "2025-06-01T10:00:00Z"
}| Field | Type | Required | Default | Description |
|---|---|---|---|---|
queue_name | string | Yes | - | Queue to place the task in: letters, digits and . _ - : @, at most 128 characters |
task_name | string | Yes | - | Human-readable task identifier |
input | JSON | No | null | Task payload (any valid JSON, up to limits.max_payload_bytes, 1 MiB by default) |
priority | integer | No | 0 | Higher = higher priority |
max_retries | integer | No | 3 | Attempt limit: a failure on this attempt dead-letters the task |
timeout_seconds | integer | No | 300 | Lease timeout per attempt |
idempotency_key | string | No | null | One task per key and queue: a repeated create returns the first task |
metadata | JSON | No | {} | Arbitrary metadata |
scheduled_at | RFC 3339 | No | null | Delayed execution time |
The response is sent once the task is durable in the bucket.
Response 201 Created:
{
"id": "01912345-6789-7abc-def0-123456789abc",
"queue_name": "emails",
"task_name": "send-welcome-email",
"status": "PENDING",
"priority": 0,
"max_retries": 3,
"attempt_count": 0,
"input": { "to": "user@example.com", "subject": "Welcome!" },
"created_at": "2025-01-15T10:00:00Z",
"updated_at": "2025-01-15T10:00:00Z"
}A create whose idempotency_key is already in use in its queue returns the task that key made, with
200 OK instead of 201 and the header idempotent-replayed: true (false on a new task). That
makes a create safe to retry after a timeout or a lost response: the SDK clients attach a
generated key (auto-…) to every create that has none for this reason.
A node at its task limit (limits.max_tasks_in_ram) answers 429 with code
RESOURCE_EXHAUSTED until workers drain the backlog.
Get a Task
GET /api/v1/tasks/{task_id}Returns the full task object including output, error_message, and timing fields.
List Tasks
GET /api/v1/tasks?queue_name=emails&status=RUNNING&limit=50&offset=0| Param | Type | Default | Description |
|---|---|---|---|
queue_name | string | - | Filter by queue |
status | string | - | Filter by status: PENDING, RUNNING, COMPLETED, FAILED, RETRY, DEAD_LETTER, CANCELLED |
limit | integer | 50 | Max results (1–1000) |
offset | integer | 0 | Pagination offset |
Newest first. Lists are served from RAM: finished tasks are listed for
completed_retention_secs (24 h by default) after they end.
Cancel a Task
POST /api/v1/tasks/{task_id}/cancelCancels a task in any non-terminal state: PENDING, RETRY or RUNNING. A running task's
worker receives a cancellation. Returns 422 INVALID_STATE for an unknown or finished task.
Delete a Task
DELETE /api/v1/tasks/{task_id}Returns { "deleted": true }, or 404 for an unknown task.
Clear All Tasks
DELETE /api/v1/tasksReturns { "deleted_count": 42 }. Runs on every node.
Signals
Send a Signal
POST /api/v1/tasks/{task_id}/signal{
"signal_name": "progress_request",
"payload": { "include_stats": true }
}Response 201 Created:
{
"signal_id": "01912345-...",
"delivered": true
}A signal can be sent to any non-terminal task. delivered is false when the task has no
running worker to hand it to yet; the signal stays PENDING and is delivered when a run starts.
List Signals
GET /api/v1/tasks/{task_id}/signals?status=PENDINGTask Runs
Get Runs
GET /api/v1/tasks/{task_id}/runsReturns all execution attempts with worker info (worker_id, assigned_node_id), lease,
timing, and output.
Get Run Logs
GET /api/v1/tasks/{task_id}/runs/{run_id}/logs?limit=1000&after_id=...Checkpoints
Get Checkpoints
GET /api/v1/tasks/{task_id}/checkpointsReturns the task's completed steps in the order they were first completed. A step
checkpointed again by a later attempt is updated in place. Returns 404 for an unknown
task.
[
{
"task_id": "0192...",
"step": "fetch",
"output": {"rows": 3},
"run_id": "0192...",
"attempt_number": 1,
"created_at": "2026-09-24T10:00:00+00:00"
}
]Workers
List Workers
GET /api/v1/workersEvery connected worker on every node.
[
{
"id": "01912345-...",
"node_id": "valka-1",
"name": "email-worker",
"queues": ["emails"],
"concurrency": 8,
"active_tasks": 3,
"status": "CONNECTED",
"last_heartbeat": "2025-01-15T10:00:30Z",
"connected_at": "2025-01-15T09:00:00Z"
}
]Dead Letter Queue
List Dead Letters
GET /api/v1/dead-letters?queue_name=emails&limit=50&offset=0Events (SSE)
Subscribe to Events
GET /api/v1/eventsServer-Sent Events stream of task status changes. Every node relays the other nodes' events, so one subscription sees the whole cluster. Each event:
data: {"event_id":"...","task_id":"...","queue_name":"emails","new_status":4,"timestamp_ms":1705312800000}new_status is the numeric TaskStatus value from the gRPC API
(4 is COMPLETED).
Cluster
Read-only views of nodes, shards and storage. They need no token.
Overview
GET /api/v1/clusterEvery node with its stats, plus nodes known only from their lease (dead, suspect or not answering), and an overall health verdict.
{
"cluster_id": "0199...",
"this_node": "valka-1",
"clustered": true,
"num_shards": 4096,
"health": {
"status": "ok",
"unowned_shards": 0,
"poisoned_nodes": [],
"suspect_nodes": [],
"partial": false
},
"nodes": [
{
"node_id": "valka-1",
"inc": 3,
"status": "alive",
"lease_age_ms": 812,
"shards_owned": 1365,
"tasks": { "pending": 3, "running": 8, "retry": 0, "completed": 120, "failed": 0, "dead_letter": 1, "cancelled": 0, "total": 132 },
"workers_connected": 2,
"wal": { "inc": 3, "durable_lsn": "3:1840", "next_lsn": "3:1841", "unflushed_records": 0, "oldest_unacked_ms": null, "poisoned": null },
"eligible": true
}
]
}Node entries also carry addresses, version, start time, storage backend, queues and snapshot
stats (trimmed above). health.status is ok, degraded (unowned shards, suspect nodes, or a
partial answer) or critical (a poisoned or self-fenced node). Node status is one of alive, draining,
fenced, poisoned for nodes that answered, and suspect, dead, draining, alive or
unknown for nodes known only from their lease. unowned_shards is null when the answer is
partial.
Members
GET /api/v1/cluster/membersThis node's view of membership, straight from the leases in the bucket: every node id ever seen,
with inc, status, state (alive, draining, dead), live, suspect, eligible,
lease_age_ms, addresses, version, protocol, started_at, unsplit (earlier
incarnations whose log is not split yet) and dead_by (for a dead lease: the guardian that
tombstoned it, or the node itself after a clean exit).
Shards
GET /api/v1/cluster/shards?node=valka-2&dirty=true&min_tasks=1One row per shard, gathered from every node: shard, owner (null when nobody serves it right
now), gen (ownership generation), phase (unowned, loading, serving, releasing), task
and signal counts, shard_seq, the latest snapshot and records_since_snapshot.
| Param | Description |
|---|---|
node | Only shards owned by this node |
dirty | true: only shards with records not covered by a snapshot |
min_tasks | Only shards holding at least this many tasks |
unowned | true: only shards nobody serves right now |
One Shard
GET /api/v1/cluster/shards/{shard}A shard row plus its task counts per queue, from the node serving it. 404 for a shard number
out of range.
Storage
GET /api/v1/cluster/storageObject counts and sizes in the bucket: this node's WAL segments, all snapshots and task log chunks, with the age of the oldest objects, plus an upper-bound estimate of daily PUT requests and their S3 Standard cost. Cached for 30 s.
Snapshot Now
POST /api/v1/cluster/snapshotSnapshots the dirty shards of the node that received the request, now. Returns its snapshot stats and durable LSN.
Placement
GET /api/v1/cluster/placement{
"pins": { "42": "valka-1" },
"effective_protocol": 2,
"eligible_nodes": ["valka-1", "valka-2", "valka-3"]
}pins are operator pins (shard to node). effective_protocol is the lowest protocol version
among live nodes.
Queues
GET /api/v1/queuesEvery queue's partition config: queue, partitions (new tasks go to these),
served_partitions (every partition the queue ever had: after a shrink, the removed ones take
no new tasks but are served until empty), previous (the count before the last change, if
any), version, created_at, and partition_shards (each served partition's shard, desired
owner and draining flag).
Admin
The admin endpoints change where things live. They are disabled until VALKA_ADMIN_TOKEN is
set, and need Authorization: Bearer <admin token>. Every action is written to the bucket as an
audit record. See Clustering for when to use them.
| Status | Code | Meaning |
|---|---|---|
| 403 | FORBIDDEN | The admin API is disabled (no VALKA_ADMIN_TOKEN) |
| 401 | UNAUTHORIZED | Bad or missing admin token |
Drain a Node
POST /api/v1/cluster/nodes/{node}/drainThe node hands every shard over with final snapshots, marks its lease dead and exits.
Response 202 Accepted: { "accepted": true }. 404 if no live node has that id, 503 if
the node did not accept.
Move (Pin) a Shard
POST /api/v1/cluster/shards/{shard}/move{ "to": "valka-1" }Pins the shard to an eligible node; its owner hands it over on its next placement round. The pin holds while that node is eligible.
Response 202 Accepted:
{ "shard": 42, "pinned_to": "valka-1", "placement_version": 3 }400 if the target is not an eligible node (live, alive, past its join delay), 404 for a shard
out of range.
Unpin a Shard
DELETE /api/v1/cluster/shards/{shard}/pinBack to automatic placement. Returns { "shard": 42, "placement_version": 4 }.
Set a Queue's Partitions
PUT /api/v1/queues/{queue}/partitions{ "partitions": 32 }Between 1 and 4096. Growing is online: new partitions take new tasks while old ones drain. Keyed creates also check a key's old partition while the change is recent, so idempotency stays exact.
Response 200 OK:
{
"queue": "emails",
"partitions": 32,
"previous": 16,
"version": 2,
"nodes_not_told": []
}nodes_not_told lists nodes that could not be told to drop their cached copy; they pick up the
change within 30 s anyway.
Audit Log
GET /api/v1/cluster/audit?limit=50The newest admin actions, newest first (limit 1–500). Each record has id, action
(drain_node, move_shard, unpin_shard, set_partitions), params, result, node (the node
that ran it) and at.
Monitoring
Health Check
GET /healthzReturns "ok" with status 200, or 503 with "wal writer poisoned" when the node can no
longer write to the bucket (it exits right after).
Prometheus Metrics
GET /metricsReturns metrics in Prometheus text format. See Deployment.
Error Responses
All errors follow this format:
{
"error": "Task not found",
"code": "NOT_FOUND"
}| Status | Code | Description |
|---|---|---|
| 400 | BAD_REQUEST | Invalid argument (missing field, unknown status, bad timestamp, payload too large) |
| 401 | UNAUTHENTICATED | Missing or wrong API key |
| 401 | UNAUTHORIZED | Missing or wrong admin token (admin endpoints) |
| 403 | FORBIDDEN | Admin API disabled: the server has no VALKA_ADMIN_TOKEN |
| 404 | NOT_FOUND | Resource not found |
| 422 | INVALID_STATE | Task in invalid state for the operation |
| 429 | RESOURCE_EXHAUSTED | The node holds as many tasks as its limit allows. Retry later |
| 503 | UNAVAILABLE | The shard is moving, its owner is unreachable, or the bucket is failing. Retry |
| 500 | INTERNAL_ERROR | Server error |