valka

gRPC API

gRPC service definitions and streaming protocols.

Valka's gRPC API runs on port 50051 by default. Proto files are at proto/valka/v1/.

Every node serves ApiService and WorkerService. In a cluster, a call about a task is forwarded to the node that owns the task's shard, and a worker connected to any node can run tasks owned by any node.

Status Codes

CodeWhen
NOT_FOUNDUnknown task
INVALID_ARGUMENTMissing or invalid field
ALREADY_EXISTSThe idempotency key is already used
FAILED_PRECONDITIONThe task is in the wrong state for the call (e.g. cancelling a finished task)
UNAVAILABLEThe shard is moving, its owner is unreachable, or the bucket is failing. Retry
INTERNALServer error

ApiService

The main client-facing API for task management.

service ApiService {
    rpc CreateTask(CreateTaskRequest) returns (CreateTaskResponse);
    rpc GetTask(GetTaskRequest) returns (GetTaskResponse);
    rpc ListTasks(ListTasksRequest) returns (ListTasksResponse);
    rpc CancelTask(CancelTaskRequest) returns (CancelTaskResponse);
    rpc SendSignal(SendSignalRequest) returns (SendSignalResponse);
    rpc SubscribeEvents(SubscribeEventsRequest) returns (stream TaskEvent);
    rpc SubscribeLogs(SubscribeLogsRequest) returns (stream LogEntry);
}

CreateTask

Create a new task in a queue.

FieldTypeDescription
queue_namestringTarget queue
task_namestringTask identifier
inputstring (JSON)Task payload
priorityint32Priority (higher = first)
max_retriesint32Max retry attempts
timeout_secondsint32Lease timeout
idempotency_keystringDedup key
metadatastring (JSON)Arbitrary metadata
scheduled_atTimestampDelayed execution

SubscribeEvents

Server-streaming RPC. Returns a stream of TaskEvent messages for real-time monitoring.

message TaskEvent {
    string event_id = 1;
    string task_id = 2;
    string queue_name = 3;
    TaskStatus new_status = 4;
    int64 timestamp_ms = 5;
}

SubscribeLogs

Server-streaming RPC. Returns a stream of LogEntry messages for a specific task run.

WorkerService

The bidirectional streaming service used by workers.

service WorkerService {
    rpc Session(stream WorkerRequest) returns (stream WorkerResponse);
    // Persist the result of a completed step of a running task. Returns once durable.
    rpc Checkpoint(CheckpointRequest) returns (CheckpointResponse);
}

A single Session stream carries all communication between a worker and the server. Checkpoint is a separate unary call so a handler can wait for its step to be durable.

Worker → Server Messages

graph LR
  subgraph WorkerRequest
      WH[WorkerHello<br/>Initial handshake]
      TR[TaskResult<br/>Success or failure]
      HB[Heartbeat<br/>Lease extension]
      LB[LogBatch<br/>Log entries]
      GS[GracefulShutdown<br/>Drain signal]
      SA[SignalAck<br/>Signal confirmed]
  end
MessageWhen SentDescription
WorkerHelloOn connectWorker name, queues, concurrency, and the runs it is still executing (running), so they are reattached after a reconnect
TaskResultTask doneSuccess/failure with output/error; resent until a ResultAck says APPLIED or STALE
HeartbeatEvery 10sIDs of tasks running or awaiting a result ack, for lease extension
LogBatchDuring taskStructured log entries
GracefulShutdownShutting downStop sending tasks; the stream stays open until in-flight results are acknowledged
SignalAckSignal receivedConfirms signal delivery

Server → Worker Messages

MessageWhen SentDescription
TaskAssignmentTask matchedNew task to execute, with checkpoints from earlier attempts
TaskCancellationCancel requestCancel a running task. Also sent for a run in the hello that has already ended
HeartbeatAckAfter heartbeatConfirms heartbeat received
ServerShutdownServer stoppingThe stream is about to close; the worker reconnects with backoff
TaskSignalSignal sentReal-time signal for a task
ResultAckAfter a TaskResultAPPLIED, STALE or RETRY (see below)

Result acknowledgement

Every TaskResult is answered with a ResultAck:

StatusMeaningWorker action
RESULT_STATUS_APPLIEDThe result is durable, now or from an earlier delivery of the same resultForget it
RESULT_STATUS_STALEThe run already ended another way (lease expired, cancelled, deleted); the result was not recordedForget it
RESULT_STATUS_RETRYNot recorded yet (storage trouble)Send it again later

Results are idempotent: sending the same result twice is safe. Every SDK keeps a result until it is APPLIED or STALE, resends it after a reconnect, and keeps heartbeating the task meanwhile, so a handler that finishes while the server is restarting still has its result recorded, without the task running again.

Step checkpoints

message CheckpointRequest {
    string task_id = 1;
    string task_run_id = 2;
    string step = 3;
    string output = 4;   // JSON
}

Only the task's current running run may checkpoint. The call returns once the checkpoint is durable. Every later TaskAssignment for the task carries its checkpoints (repeated StepCheckpoint checkpoints), which the SDKs' step helpers use to skip steps an earlier attempt completed. Limits: step names up to 256 bytes, outputs up to 256 KiB of JSON, at most 256 distinct steps per task.

Reconnect and reattach

WorkerHello.running lists the runs a reconnecting worker is still executing:

message RunningTask {
    string task_id = 1;
    string task_run_id = 2;
    int32 attempt_number = 3;
}

For each one, the task's owner reattaches the run to the new connection, adopts it if its dispatch record was lost in a crash, or answers with a TaskCancellation if the run is stale. The worker also resends every result not yet acknowledged.

InternalService

Node-to-node calls in a cluster, on the internal listener (cluster.internal_addr, default :50052). Every call carries the cluster token (authorization: Bearer <token>), optionally over mTLS. This service is not for clients; never expose its port publicly.

service InternalService {
    // Run an id-routed command on the node that owns its shard.
    rpc Execute(ExecuteRequest) returns (ExecuteResponse);
    // Answer a cluster-wide query from this node's shards only (list, count, stats).
    rpc Scatter(ScatterRequest) returns (ScatterResponse);
    // This node's task events, for nodes whose subscribers want the whole cluster's stream.
    rpc Events(EventsRequest) returns (stream TaskEvent);
    // Health and identity.
    rpc Ping(PingRequest) returns (PingResponse);
}

A node that does not own the shard of an Execute command answers NOT_OWNER with a hint of the owner it knows; the caller retries there.

Common Types

TaskStatus

enum TaskStatus {
    TASK_STATUS_UNSPECIFIED = 0;
    TASK_STATUS_PENDING = 1;
    TASK_STATUS_DISPATCHING = 2;
    TASK_STATUS_RUNNING = 3;
    TASK_STATUS_COMPLETED = 4;
    TASK_STATUS_FAILED = 5;
    TASK_STATUS_RETRY = 6;
    TASK_STATUS_DEAD_LETTER = 7;
    TASK_STATUS_CANCELLED = 8;
}

TASK_STATUS_DISPATCHING is kept for compatibility but never used: a dispatched task is RUNNING.

LogLevel

enum LogLevel {
    LOG_LEVEL_UNSPECIFIED = 0;
    LOG_LEVEL_DEBUG = 1;
    LOG_LEVEL_INFO = 2;
    LOG_LEVEL_WARN = 3;
    LOG_LEVEL_ERROR = 4;
}

Using with grpcurl

# Create a task
grpcurl -plaintext -d '{
  "queue_name": "emails",
  "task_name": "send-email",
  "input": "{\"to\": \"user@example.com\"}"
}' localhost:50051 valka.v1.ApiService/CreateTask

# Subscribe to events
grpcurl -plaintext localhost:50051 valka.v1.ApiService/SubscribeEvents

On this page