Task Lifecycle
Understanding task states, transitions, retries, and signals.
Task States
Every task in Valka moves through a well-defined state machine. Each arrow is a record in the
write-ahead log (TaskCreated, TaskDispatched, RunCompleted, RunFailed, LeaseExpired,
TaskPromoted, TaskCancelled, ...), so a task's state survives any node failure.
stateDiagram-v2
[*] --> PENDING: Task created
PENDING --> RUNNING: Dispatched to a worker
RUNNING --> PENDING: Run never reached its worker
RUNNING --> COMPLETED: Success
RUNNING --> FAILED: Non-retryable error
RUNNING --> RETRY: Retryable error or lease expired
RUNNING --> DEAD_LETTER: Out of attempts
RETRY --> PENDING: Backoff elapsed
PENDING --> CANCELLED: Cancel API
RETRY --> CANCELLED: Cancel API
RUNNING --> CANCELLED: Cancel API
COMPLETED --> [*]
FAILED --> [*]
DEAD_LETTER --> [*]
CANCELLED --> [*]
State Descriptions
| State | Description |
|---|---|
PENDING | Waiting for a worker. A scheduled task is PENDING but not dispatched before its time |
RUNNING | A worker is processing the task. The run holds a lease |
COMPLETED | Task finished successfully |
FAILED | Task failed with a non-retryable error |
RETRY | Task failed (or its lease expired) and has attempts left. It returns to PENDING after a backoff |
DEAD_LETTER | Task used all its attempts |
CANCELLED | Task was cancelled via the API |
A task can be cancelled from any non-terminal state: PENDING, RETRY or RUNNING.
A run that the owner recorded but that never reached its worker (the worker vanished, or its node
could not deliver the assignment) goes back to PENDING without spending an attempt.
The protobuf TaskStatus enum still has a DISPATCHING value, but no task is ever stored in that
state: a dispatched task is RUNNING.
Retries
When a task handler returns an error, Valka retries it with exponential backoff:
- Attempts:
max_retries(default 3) caps the attempts. When the attempt number reachesmax_retries, the next failure moves the task toDEAD_LETTER. - Backoff: the delay after attempt
nfails isretry_base_delay_secs × 2^n, capped atretry_max_delay_secs. With the defaults (1 s and 3600 s) that is 2 s, 4 s, 8 s, ... - Non-retryable errors: a handler can report its error as not retryable. The task then
becomes
FAILEDat once, unless it was already on its last attempt (thenDEAD_LETTER). In the Go, TypeScript and Python SDKs, fail with aHandlerErrorwhoseretryableis false; the Rust SDK treats every handler error as retryable.
sequenceDiagram
participant W as Worker
participant O as Owner node
participant B as Bucket
W->>O: TaskResult (retryable error)
O->>B: RunFailed (RETRY, next attempt at now + backoff)
O-->>W: ResultAck APPLIED
Note over O: Timer fires when the backoff elapses
O->>B: TaskPromoted (PENDING)
O->>W: TaskAssignment (attempt 2)
W->>O: TaskResult (success)
O->>B: RunCompleted (COMPLETED)
O-->>W: ResultAck APPLIED
Retry timers run on the node that owns the task's shard, checked every 100 ms. They are not stored separately: after a restart or a takeover they are rebuilt from the tasks' state.
Dead Letter Queue
When a task exhausts its attempts, it moves to DEAD_LETTER and gets a dead-letter entry in its
shard. Dead letter tasks can be:
- Inspected via the REST API (
GET /api/v1/dead-letters) or the dashboard - Replayed manually by creating a new task with the same input
Lease Management
When a task moves to RUNNING, its run gets a lease: the task's timeout plus a 30 s grace.
Workers heartbeat every 10 s, and each heartbeat keeps the lease at least 60 s ahead.
If a worker crashes or disconnects:
- The owner's timer fires when the lease expires (checked every 100 ms)
- The task moves to
RETRY(if attempts remain) orDEAD_LETTER - Another worker can pick up the retried task
sequenceDiagram
participant W as Worker
participant S as Owner node
S->>W: TaskAssignment (lease: timeout + 30 s)
loop Every 10 s
W->>S: Heartbeat
S->>S: Lease at least now + 60 s
end
Note over W: Worker crashes!
Note over S: Lease timer fires
S->>S: LeaseExpired, RETRY or DEAD_LETTER
Note over S: Task available for another worker
If the worker's node dies instead, the worker keeps running the task, reconnects and
reattaches it, so the task is not retried. After a takeover, recovered RUNNING tasks get a 60 s
grace lease so their workers have time to reconnect.
Result Delivery
A worker's result is acknowledged end to end. The server answers every result with
APPLIED (recorded, now or by an earlier delivery), STALE (the run already ended another
way, e.g. its lease expired) or RETRY. The SDKs keep each result until it is APPLIED or
STALE and keep heartbeating the task meanwhile, so:
- a result is never lost because the server was busy, restarting, or briefly unreachable;
- a task whose handler finished during a server restart completes once, without re-running;
- sending the same result twice is harmless.
In a cluster, the worker's node forwards the result to the task's owner, wherever it is, and relays the acknowledgement back.
Step Checkpoints
A handler can split its work into named steps. Each step's result is checkpointed before the step returns, and every retry of the task (after a failure, a lease expiry or a server crash) receives the recorded results, so completed steps are skipped:
let rows: Vec<Row> = ctx.step("fetch", || fetch_rows()).await?;
let report: Report = ctx.step("transform", || build_report(&rows)).await?;
ctx.step("upload", || upload(&report)).await?;A checkpointed step never runs again. The step in progress when an attempt dies does run
again (at-least-once), so steps with external side effects should still be idempotent. See
the SDK pages for each language, and GET /api/v1/tasks/{id}/checkpoints to inspect them.
Task Signals
Signals allow you to send real-time messages to running tasks. This is useful for:
- Requesting progress updates
- Sending configuration changes
- Implementing graceful cancellation
Signal Flow
sequenceDiagram
participant C as Client
participant A as Any node
participant O as Owner node
participant D as Worker node
participant W as Worker
C->>A: POST /tasks/:id/signal
A->>O: forward to the shard owner
O->>O: SignalCreated (PENDING), durable
O->>D: deliver to the run's node
D->>W: TaskSignal (gRPC stream)
O->>O: SignalDelivered (DELIVERED)
W->>D: SignalAck
D->>O: ack
O->>O: SignalAcked (ACKNOWLEDGED)
Signal States
| State | Description |
|---|---|
PENDING | Signal created, waiting for delivery |
DELIVERED | Signal sent to the worker over its gRPC stream |
ACKNOWLEDGED | Worker confirmed receipt |
A signal can be sent to any non-terminal task. A signal sent before the task runs is delivered
when its run starts. If a worker disconnects before acknowledging, unacknowledged signals reset to
PENDING for redelivery.
Sending a Signal
curl -X POST http://localhost:8989/api/v1/tasks/{task_id}/signal \
-H "Content-Type: application/json" \
-d '{"signal_name": "progress_request", "payload": {}}'Scheduled Tasks
Tasks can be created with a scheduled_at timestamp (RFC 3339) for delayed execution:
curl -X POST http://localhost:8989/api/v1/tasks \
-H "Content-Type: application/json" \
-d '{
"task_name": "send-reminder",
"queue_name": "emails",
"input": {"to": "user@example.com"},
"scheduled_at": "2026-01-15T10:00:00Z"
}'The task stays in PENDING state but is not eligible for dispatch until the scheduled time. The
owner's timer then promotes it (TaskPromoted) and it is offered to workers.
Idempotency
Tasks support idempotency keys to prevent duplicates:
curl -X POST http://localhost:8989/api/v1/tasks \
-H "Content-Type: application/json" \
-d '{
"task_name": "charge-payment",
"queue_name": "payments",
"input": {"order_id": "abc-123", "amount": 99.99},
"idempotency_key": "payment-abc-123"
}'If a task with the same idempotency key already exists in the same queue, no second task is made: the create
returns the existing task (REST 200 OK with idempotent-replayed: true instead of 201). So a
create is safe to retry after a timeout, and the SDK clients give every keyless create a
generated key for exactly that reason. A key always maps to the same queue partition,
so this holds on every node of a cluster, and also after the queue's partition count changes. A
key is remembered as long as its task is held in RAM: while it is live, and for
completed_retention_secs (24 h by default) after it finishes.