valka
SDKs

Python SDK

Build workers and manage tasks with the Python SDK.

Installation

pip install "git+https://github.com/IWhitebird/Valka@v0.2.0#subdirectory=sdks/python"

The package is not on PyPI yet; pip builds it from the repository. Python 3.10 or newer.

The Python SDK provides:

  • ValkaWorker: process tasks from queues
  • ValkaClient: create, get, list, and cancel tasks
  • Full async/await support

Worker

import asyncio
from valka import ValkaWorker, TaskContext

async def handle_task(ctx: TaskContext) -> dict:
    data = ctx.input()
    to = data.get("to", "unknown") if data else "unknown"

    await ctx.log(f"Processing email task: {ctx.task_name}")
    await ctx.debug(f"Attempt #{ctx.attempt_number}")

    # Simulate work
    await asyncio.sleep(1)

    await ctx.log(f"Email sent to {to}")
    return {"delivered_to": to, "status": "sent"}

async def main():
    worker = (
        ValkaWorker.builder()
        .name("email-worker")
        .server_addr("localhost:50051")
        .queues(["emails", "notifications"])
        .concurrency(4)
        .handler(handle_task)
        .build()
    )
    await worker.run()

if __name__ == "__main__":
    asyncio.run(main())

Builder Options

MethodDescription
.name(name)Worker name
.server_addr(addr)Valka server gRPC address
.server_addrs([...])Several nodes; the worker moves to the next when its connection drops
.api_key(key)API key (default: VALKA_API_KEY)
.queues([...])Queues to listen on
.concurrency(n)Max concurrent tasks
.handler(fn)Async function to process tasks

Task Context

async def handle_task(ctx: TaskContext) -> dict:
    # Access metadata
    name = ctx.task_name
    attempt = ctx.attempt_number
    task_id = ctx.task_id

    # Parse input (returns dict)
    data = ctx.input()

    # Logging
    await ctx.log("Processing...")
    await ctx.debug("Debug info")

    # Return output (any JSON-serializable dict)
    return {"result": "done"}

Step Checkpoints

ctx.step(name, fn) runs fn once per task: its result is checkpointed, and a retry of the task returns the recorded result instead of running fn again. fn may be sync or async.

async def handle_task(ctx: TaskContext) -> dict:
    rows = await ctx.step("fetch", fetch_rows)
    report = await ctx.step("transform", lambda: build_report(rows))
    await ctx.step("upload", lambda: upload(report))
    return {"rows": len(rows)}

Step results must be JSON-serializable. await ctx.checkpoint(step, value) and ctx.checkpoint_value(step, default) record and read checkpoints directly.

Client

The client uses the REST API (port 8989).

import asyncio
from valka import ValkaClient

async def main():
    async with ValkaClient(
        "http://valka-1:8989",
        base_urls=["http://valka-2:8989", "http://valka-3:8989"],  # optional: more nodes
        api_key="...",                                              # or VALKA_API_KEY
    ) as client:
        task = await client.create_task(
            "emails",
            "send-email",
            input={"to": "user@example.com", "subject": "Welcome!"},
            max_retries=3,
        )
        print("created", task["id"])

        task = await client.get_task(task["id"])
        print("status", task["status"])

asyncio.run(main())

Several nodes, API keys and retries

  • Give the client every node's URL. A call that fails because a node is unreachable or busy (connection errors, 429, 502, 503, 504) is retried with backoff on the next node, up to attempts times (default 6).
  • Every create carries an idempotency key: yours, or a generated auto-… one. A retried create therefore returns the task the first attempt made instead of making a second. Pass auto_idempotency=False to turn this off.
  • Cancels and signals are sent once, since repeating them is not safe.
  • api_key (default: the VALKA_API_KEY environment variable) is sent as Authorization: Bearer <key>.

Signal Handling

worker = (
    ValkaWorker.builder()
    .name("worker")
    .server_addr("localhost:50051")
    .queues(["emails"])
    .concurrency(4)
    .handler(handle_task)
    .on_signal(handle_signal)
    .build()
)

async def handle_signal(signal):
    print(f"Signal received: {signal.signal_type}")

Reconnects

The worker survives server restarts and node failures without losing work:

  • When its stream breaks (or the server sends ServerShutdown), the worker reconnects with exponential backoff, from 100 ms up to 30 s, to its node (the next one in its list when it has several). Running handlers keep going meanwhile.
  • On the new connection it resends every result not yet acknowledged (APPLIED or STALE), and keeps heartbeating those tasks until then.
  • Its hello lists the runs it is still executing. The task's owner reattaches each one to the new connection instead of running the task again, or records a dispatch that was lost in a crash. A run that ended meanwhile (cancelled, expired) gets a cancellation.

In a cluster, give the worker every node with .server_addrs([...]) (it moves to the next on each reconnect), or point it at a load balancer or DNS name that covers several nodes, so it lands on a live node when its own dies. See Clustering.

On this page