valka

REST API

Complete REST API reference for task management, signals, monitoring, the cluster and the admin API.

The Valka REST API runs on port 8989 by default. All endpoints return JSON.

In a cluster, every node serves the whole API. A request about one task is forwarded to the node that owns the task's shard; a list is gathered from every live node and merged. You can send any request to any node.

Authentication

When the server is started with VALKA_API_KEYS (comma-separated), every /api/v1 endpoint needs one of those keys:

Authorization: Bearer <api key>

A missing or wrong key gets 401 with code UNAUTHENTICATED. The event stream also accepts the key as ?access_token=<key>, because a browser's EventSource cannot send headers. /healthz and /metrics stay open for probes and scrapers. The admin endpoints use the separate admin token instead. Without VALKA_API_KEYS the API is open, and the server logs a warning at startup.

Partial answers

Endpoints that gather from every node (GET /api/v1/tasks, DELETE /api/v1/tasks, GET /api/v1/workers, GET /api/v1/dead-letters) set the X-Valka-Partial response header:

X-Valka-Partial: false
X-Valka-Partial: true; missing=valka-2

true means some live node did not answer in time, and the list is missing its part. Retry, or check the cluster.

Tasks

Create a Task

POST /api/v1/tasks
{
  "queue_name": "emails",
  "task_name": "send-welcome-email",
  "input": { "to": "user@example.com", "subject": "Welcome!" },
  "priority": 0,
  "max_retries": 3,
  "timeout_seconds": 300,
  "idempotency_key": "welcome-user-123",
  "metadata": { "source": "signup-flow" },
  "scheduled_at": "2025-06-01T10:00:00Z"
}
FieldTypeRequiredDefaultDescription
queue_namestringYes-Queue to place the task in: letters, digits and . _ - : @, at most 128 characters
task_namestringYes-Human-readable task identifier
inputJSONNonullTask payload (any valid JSON, up to limits.max_payload_bytes, 1 MiB by default)
priorityintegerNo0Higher = higher priority
max_retriesintegerNo3Attempt limit: a failure on this attempt dead-letters the task
timeout_secondsintegerNo300Lease timeout per attempt
idempotency_keystringNonullOne task per key and queue: a repeated create returns the first task
metadataJSONNo{}Arbitrary metadata
scheduled_atRFC 3339NonullDelayed execution time

The response is sent once the task is durable in the bucket.

Response 201 Created:

{
  "id": "01912345-6789-7abc-def0-123456789abc",
  "queue_name": "emails",
  "task_name": "send-welcome-email",
  "status": "PENDING",
  "priority": 0,
  "max_retries": 3,
  "attempt_count": 0,
  "input": { "to": "user@example.com", "subject": "Welcome!" },
  "created_at": "2025-01-15T10:00:00Z",
  "updated_at": "2025-01-15T10:00:00Z"
}

A create whose idempotency_key is already in use in its queue returns the task that key made, with 200 OK instead of 201 and the header idempotent-replayed: true (false on a new task). That makes a create safe to retry after a timeout or a lost response: the SDK clients attach a generated key (auto-…) to every create that has none for this reason.

A node at its task limit (limits.max_tasks_in_ram) answers 429 with code RESOURCE_EXHAUSTED until workers drain the backlog.

Get a Task

GET /api/v1/tasks/{task_id}

Returns the full task object including output, error_message, and timing fields.

List Tasks

GET /api/v1/tasks?queue_name=emails&status=RUNNING&limit=50&offset=0
ParamTypeDefaultDescription
queue_namestring-Filter by queue
statusstring-Filter by status: PENDING, RUNNING, COMPLETED, FAILED, RETRY, DEAD_LETTER, CANCELLED
limitinteger50Max results (1–1000)
offsetinteger0Pagination offset

Newest first. Lists are served from RAM: finished tasks are listed for completed_retention_secs (24 h by default) after they end.

Cancel a Task

POST /api/v1/tasks/{task_id}/cancel

Cancels a task in any non-terminal state: PENDING, RETRY or RUNNING. A running task's worker receives a cancellation. Returns 422 INVALID_STATE for an unknown or finished task.

Delete a Task

DELETE /api/v1/tasks/{task_id}

Returns { "deleted": true }, or 404 for an unknown task.

Clear All Tasks

DELETE /api/v1/tasks

Returns { "deleted_count": 42 }. Runs on every node.

Signals

Send a Signal

POST /api/v1/tasks/{task_id}/signal
{
  "signal_name": "progress_request",
  "payload": { "include_stats": true }
}

Response 201 Created:

{
  "signal_id": "01912345-...",
  "delivered": true
}

A signal can be sent to any non-terminal task. delivered is false when the task has no running worker to hand it to yet; the signal stays PENDING and is delivered when a run starts.

List Signals

GET /api/v1/tasks/{task_id}/signals?status=PENDING

Task Runs

Get Runs

GET /api/v1/tasks/{task_id}/runs

Returns all execution attempts with worker info (worker_id, assigned_node_id), lease, timing, and output.

Get Run Logs

GET /api/v1/tasks/{task_id}/runs/{run_id}/logs?limit=1000&after_id=...

Checkpoints

Get Checkpoints

GET /api/v1/tasks/{task_id}/checkpoints

Returns the task's completed steps in the order they were first completed. A step checkpointed again by a later attempt is updated in place. Returns 404 for an unknown task.

[
  {
    "task_id": "0192...",
    "step": "fetch",
    "output": {"rows": 3},
    "run_id": "0192...",
    "attempt_number": 1,
    "created_at": "2026-09-24T10:00:00+00:00"
  }
]

Workers

List Workers

GET /api/v1/workers

Every connected worker on every node.

[
  {
    "id": "01912345-...",
    "node_id": "valka-1",
    "name": "email-worker",
    "queues": ["emails"],
    "concurrency": 8,
    "active_tasks": 3,
    "status": "CONNECTED",
    "last_heartbeat": "2025-01-15T10:00:30Z",
    "connected_at": "2025-01-15T09:00:00Z"
  }
]

Dead Letter Queue

List Dead Letters

GET /api/v1/dead-letters?queue_name=emails&limit=50&offset=0

Events (SSE)

Subscribe to Events

GET /api/v1/events

Server-Sent Events stream of task status changes. Every node relays the other nodes' events, so one subscription sees the whole cluster. Each event:

data: {"event_id":"...","task_id":"...","queue_name":"emails","new_status":4,"timestamp_ms":1705312800000}

new_status is the numeric TaskStatus value from the gRPC API (4 is COMPLETED).

Cluster

Read-only views of nodes, shards and storage. They need no token.

Overview

GET /api/v1/cluster

Every node with its stats, plus nodes known only from their lease (dead, suspect or not answering), and an overall health verdict.

{
  "cluster_id": "0199...",
  "this_node": "valka-1",
  "clustered": true,
  "num_shards": 4096,
  "health": {
    "status": "ok",
    "unowned_shards": 0,
    "poisoned_nodes": [],
    "suspect_nodes": [],
    "partial": false
  },
  "nodes": [
    {
      "node_id": "valka-1",
      "inc": 3,
      "status": "alive",
      "lease_age_ms": 812,
      "shards_owned": 1365,
      "tasks": { "pending": 3, "running": 8, "retry": 0, "completed": 120, "failed": 0, "dead_letter": 1, "cancelled": 0, "total": 132 },
      "workers_connected": 2,
      "wal": { "inc": 3, "durable_lsn": "3:1840", "next_lsn": "3:1841", "unflushed_records": 0, "oldest_unacked_ms": null, "poisoned": null },
      "eligible": true
    }
  ]
}

Node entries also carry addresses, version, start time, storage backend, queues and snapshot stats (trimmed above). health.status is ok, degraded (unowned shards, suspect nodes, or a partial answer) or critical (a poisoned or self-fenced node). Node status is one of alive, draining, fenced, poisoned for nodes that answered, and suspect, dead, draining, alive or unknown for nodes known only from their lease. unowned_shards is null when the answer is partial.

Members

GET /api/v1/cluster/members

This node's view of membership, straight from the leases in the bucket: every node id ever seen, with inc, status, state (alive, draining, dead), live, suspect, eligible, lease_age_ms, addresses, version, protocol, started_at, unsplit (earlier incarnations whose log is not split yet) and dead_by (for a dead lease: the guardian that tombstoned it, or the node itself after a clean exit).

Shards

GET /api/v1/cluster/shards?node=valka-2&dirty=true&min_tasks=1

One row per shard, gathered from every node: shard, owner (null when nobody serves it right now), gen (ownership generation), phase (unowned, loading, serving, releasing), task and signal counts, shard_seq, the latest snapshot and records_since_snapshot.

ParamDescription
nodeOnly shards owned by this node
dirtytrue: only shards with records not covered by a snapshot
min_tasksOnly shards holding at least this many tasks
unownedtrue: only shards nobody serves right now

One Shard

GET /api/v1/cluster/shards/{shard}

A shard row plus its task counts per queue, from the node serving it. 404 for a shard number out of range.

Storage

GET /api/v1/cluster/storage

Object counts and sizes in the bucket: this node's WAL segments, all snapshots and task log chunks, with the age of the oldest objects, plus an upper-bound estimate of daily PUT requests and their S3 Standard cost. Cached for 30 s.

Snapshot Now

POST /api/v1/cluster/snapshot

Snapshots the dirty shards of the node that received the request, now. Returns its snapshot stats and durable LSN.

Placement

GET /api/v1/cluster/placement
{
  "pins": { "42": "valka-1" },
  "effective_protocol": 2,
  "eligible_nodes": ["valka-1", "valka-2", "valka-3"]
}

pins are operator pins (shard to node). effective_protocol is the lowest protocol version among live nodes.

Queues

GET /api/v1/queues

Every queue's partition config: queue, partitions (new tasks go to these), served_partitions (every partition the queue ever had: after a shrink, the removed ones take no new tasks but are served until empty), previous (the count before the last change, if any), version, created_at, and partition_shards (each served partition's shard, desired owner and draining flag).

Admin

The admin endpoints change where things live. They are disabled until VALKA_ADMIN_TOKEN is set, and need Authorization: Bearer <admin token>. Every action is written to the bucket as an audit record. See Clustering for when to use them.

StatusCodeMeaning
403FORBIDDENThe admin API is disabled (no VALKA_ADMIN_TOKEN)
401UNAUTHORIZEDBad or missing admin token

Drain a Node

POST /api/v1/cluster/nodes/{node}/drain

The node hands every shard over with final snapshots, marks its lease dead and exits. Response 202 Accepted: { "accepted": true }. 404 if no live node has that id, 503 if the node did not accept.

Move (Pin) a Shard

POST /api/v1/cluster/shards/{shard}/move
{ "to": "valka-1" }

Pins the shard to an eligible node; its owner hands it over on its next placement round. The pin holds while that node is eligible.

Response 202 Accepted:

{ "shard": 42, "pinned_to": "valka-1", "placement_version": 3 }

400 if the target is not an eligible node (live, alive, past its join delay), 404 for a shard out of range.

Unpin a Shard

DELETE /api/v1/cluster/shards/{shard}/pin

Back to automatic placement. Returns { "shard": 42, "placement_version": 4 }.

Set a Queue's Partitions

PUT /api/v1/queues/{queue}/partitions
{ "partitions": 32 }

Between 1 and 4096. Growing is online: new partitions take new tasks while old ones drain. Keyed creates also check a key's old partition while the change is recent, so idempotency stays exact.

Response 200 OK:

{
  "queue": "emails",
  "partitions": 32,
  "previous": 16,
  "version": 2,
  "nodes_not_told": []
}

nodes_not_told lists nodes that could not be told to drop their cached copy; they pick up the change within 30 s anyway.

Audit Log

GET /api/v1/cluster/audit?limit=50

The newest admin actions, newest first (limit 1–500). Each record has id, action (drain_node, move_shard, unpin_shard, set_partitions), params, result, node (the node that ran it) and at.

Monitoring

Health Check

GET /healthz

Returns "ok" with status 200, or 503 with "wal writer poisoned" when the node can no longer write to the bucket (it exits right after).

Prometheus Metrics

GET /metrics

Returns metrics in Prometheus text format. See Deployment.

Error Responses

All errors follow this format:

{
  "error": "Task not found",
  "code": "NOT_FOUND"
}
StatusCodeDescription
400BAD_REQUESTInvalid argument (missing field, unknown status, bad timestamp, payload too large)
401UNAUTHENTICATEDMissing or wrong API key
401UNAUTHORIZEDMissing or wrong admin token (admin endpoints)
403FORBIDDENAdmin API disabled: the server has no VALKA_ADMIN_TOKEN
404NOT_FOUNDResource not found
422INVALID_STATETask in invalid state for the operation
429RESOURCE_EXHAUSTEDThe node holds as many tasks as its limit allows. Retry later
503UNAVAILABLEThe shard is moving, its owner is unreachable, or the bucket is failing. Retry
500INTERNAL_ERRORServer error

On this page