valka

Task Lifecycle

Understanding task states, transitions, retries, and signals.

Task States

Every task in Valka moves through a well-defined state machine. Each arrow is a record in the write-ahead log (TaskCreated, TaskDispatched, RunCompleted, RunFailed, LeaseExpired, TaskPromoted, TaskCancelled, ...), so a task's state survives any node failure.

stateDiagram-v2
  [*] --> PENDING: Task created
  PENDING --> RUNNING: Dispatched to a worker
  RUNNING --> PENDING: Run never reached its worker
  RUNNING --> COMPLETED: Success
  RUNNING --> FAILED: Non-retryable error
  RUNNING --> RETRY: Retryable error or lease expired
  RUNNING --> DEAD_LETTER: Out of attempts
  RETRY --> PENDING: Backoff elapsed

  PENDING --> CANCELLED: Cancel API
  RETRY --> CANCELLED: Cancel API
  RUNNING --> CANCELLED: Cancel API

  COMPLETED --> [*]
  FAILED --> [*]
  DEAD_LETTER --> [*]
  CANCELLED --> [*]

State Descriptions

StateDescription
PENDINGWaiting for a worker. A scheduled task is PENDING but not dispatched before its time
RUNNINGA worker is processing the task. The run holds a lease
COMPLETEDTask finished successfully
FAILEDTask failed with a non-retryable error
RETRYTask failed (or its lease expired) and has attempts left. It returns to PENDING after a backoff
DEAD_LETTERTask used all its attempts
CANCELLEDTask was cancelled via the API

A task can be cancelled from any non-terminal state: PENDING, RETRY or RUNNING.

A run that the owner recorded but that never reached its worker (the worker vanished, or its node could not deliver the assignment) goes back to PENDING without spending an attempt.

The protobuf TaskStatus enum still has a DISPATCHING value, but no task is ever stored in that state: a dispatched task is RUNNING.

Retries

When a task handler returns an error, Valka retries it with exponential backoff:

  • Attempts: max_retries (default 3) caps the attempts. When the attempt number reaches max_retries, the next failure moves the task to DEAD_LETTER.
  • Backoff: the delay after attempt n fails is retry_base_delay_secs × 2^n, capped at retry_max_delay_secs. With the defaults (1 s and 3600 s) that is 2 s, 4 s, 8 s, ...
  • Non-retryable errors: a handler can report its error as not retryable. The task then becomes FAILED at once, unless it was already on its last attempt (then DEAD_LETTER). In the Go, TypeScript and Python SDKs, fail with a HandlerError whose retryable is false; the Rust SDK treats every handler error as retryable.
sequenceDiagram
  participant W as Worker
  participant O as Owner node
  participant B as Bucket

  W->>O: TaskResult (retryable error)
  O->>B: RunFailed (RETRY, next attempt at now + backoff)
  O-->>W: ResultAck APPLIED

  Note over O: Timer fires when the backoff elapses
  O->>B: TaskPromoted (PENDING)

  O->>W: TaskAssignment (attempt 2)
  W->>O: TaskResult (success)
  O->>B: RunCompleted (COMPLETED)
  O-->>W: ResultAck APPLIED

Retry timers run on the node that owns the task's shard, checked every 100 ms. They are not stored separately: after a restart or a takeover they are rebuilt from the tasks' state.

Dead Letter Queue

When a task exhausts its attempts, it moves to DEAD_LETTER and gets a dead-letter entry in its shard. Dead letter tasks can be:

  • Inspected via the REST API (GET /api/v1/dead-letters) or the dashboard
  • Replayed manually by creating a new task with the same input

Lease Management

When a task moves to RUNNING, its run gets a lease: the task's timeout plus a 30 s grace. Workers heartbeat every 10 s, and each heartbeat keeps the lease at least 60 s ahead.

If a worker crashes or disconnects:

  1. The owner's timer fires when the lease expires (checked every 100 ms)
  2. The task moves to RETRY (if attempts remain) or DEAD_LETTER
  3. Another worker can pick up the retried task
sequenceDiagram
  participant W as Worker
  participant S as Owner node

  S->>W: TaskAssignment (lease: timeout + 30 s)

  loop Every 10 s
      W->>S: Heartbeat
      S->>S: Lease at least now + 60 s
  end

  Note over W: Worker crashes!

  Note over S: Lease timer fires
  S->>S: LeaseExpired, RETRY or DEAD_LETTER

  Note over S: Task available for another worker

If the worker's node dies instead, the worker keeps running the task, reconnects and reattaches it, so the task is not retried. After a takeover, recovered RUNNING tasks get a 60 s grace lease so their workers have time to reconnect.

Result Delivery

A worker's result is acknowledged end to end. The server answers every result with APPLIED (recorded, now or by an earlier delivery), STALE (the run already ended another way, e.g. its lease expired) or RETRY. The SDKs keep each result until it is APPLIED or STALE and keep heartbeating the task meanwhile, so:

  • a result is never lost because the server was busy, restarting, or briefly unreachable;
  • a task whose handler finished during a server restart completes once, without re-running;
  • sending the same result twice is harmless.

In a cluster, the worker's node forwards the result to the task's owner, wherever it is, and relays the acknowledgement back.

Step Checkpoints

A handler can split its work into named steps. Each step's result is checkpointed before the step returns, and every retry of the task (after a failure, a lease expiry or a server crash) receives the recorded results, so completed steps are skipped:

let rows: Vec<Row> = ctx.step("fetch", || fetch_rows()).await?;
let report: Report = ctx.step("transform", || build_report(&rows)).await?;
ctx.step("upload", || upload(&report)).await?;

A checkpointed step never runs again. The step in progress when an attempt dies does run again (at-least-once), so steps with external side effects should still be idempotent. See the SDK pages for each language, and GET /api/v1/tasks/{id}/checkpoints to inspect them.

Task Signals

Signals allow you to send real-time messages to running tasks. This is useful for:

  • Requesting progress updates
  • Sending configuration changes
  • Implementing graceful cancellation

Signal Flow

sequenceDiagram
  participant C as Client
  participant A as Any node
  participant O as Owner node
  participant D as Worker node
  participant W as Worker

  C->>A: POST /tasks/:id/signal
  A->>O: forward to the shard owner
  O->>O: SignalCreated (PENDING), durable
  O->>D: deliver to the run's node
  D->>W: TaskSignal (gRPC stream)
  O->>O: SignalDelivered (DELIVERED)
  W->>D: SignalAck
  D->>O: ack
  O->>O: SignalAcked (ACKNOWLEDGED)

Signal States

StateDescription
PENDINGSignal created, waiting for delivery
DELIVEREDSignal sent to the worker over its gRPC stream
ACKNOWLEDGEDWorker confirmed receipt

A signal can be sent to any non-terminal task. A signal sent before the task runs is delivered when its run starts. If a worker disconnects before acknowledging, unacknowledged signals reset to PENDING for redelivery.

Sending a Signal

curl -X POST http://localhost:8989/api/v1/tasks/{task_id}/signal \
  -H "Content-Type: application/json" \
  -d '{"signal_name": "progress_request", "payload": {}}'

Scheduled Tasks

Tasks can be created with a scheduled_at timestamp (RFC 3339) for delayed execution:

curl -X POST http://localhost:8989/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "send-reminder",
    "queue_name": "emails",
    "input": {"to": "user@example.com"},
    "scheduled_at": "2026-01-15T10:00:00Z"
  }'

The task stays in PENDING state but is not eligible for dispatch until the scheduled time. The owner's timer then promotes it (TaskPromoted) and it is offered to workers.

Idempotency

Tasks support idempotency keys to prevent duplicates:

curl -X POST http://localhost:8989/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{
    "task_name": "charge-payment",
    "queue_name": "payments",
    "input": {"order_id": "abc-123", "amount": 99.99},
    "idempotency_key": "payment-abc-123"
  }'

If a task with the same idempotency key already exists in the same queue, no second task is made: the create returns the existing task (REST 200 OK with idempotent-replayed: true instead of 201). So a create is safe to retry after a timeout, and the SDK clients give every keyless create a generated key for exactly that reason. A key always maps to the same queue partition, so this holds on every node of a cluster, and also after the queue's partition count changes. A key is remembered as long as its task is held in RAM: while it is live, and for completed_retention_secs (24 h by default) after it finishes.

On this page