Deployment
Run Valka in production. One node or a cluster on Docker Compose, VMs or Kubernetes, with health checks, alerts and the full configuration reference.
A Valka deployment is one or more valka-server processes and one bucket. Nodes keep nothing
on their own disks that the bucket doesn't also have, so you can replace, restart or kill them
at any time. Run every node under a supervisor that restarts it when it exits (Docker
restart: unless-stopped, systemd, Kubernetes): a node exits on purpose when it can't write to
the bucket or finds itself fenced, and a fresh process rebuilds from the bucket.
| Port | Serves | Expose to |
|---|---|---|
50051 | Public gRPC: workers, SDK clients, CLI | Workers and clients |
8989 | REST API, dashboard, SSE events, /healthz, /metrics | Clients, operators, Prometheus |
50052 | Internal node-to-node gRPC (cluster only) | Other Valka nodes only. Never public |
The public REST and gRPC APIs have no authentication of their own (only the admin endpoints need a token). Keep them on a private network or behind an authenticating proxy.
Single node
A single node can use any storage backend. It runs the same protocol as a cluster, alone.
With a local directory
The simplest setup: the local backend stores everything in one directory. The Docker image
defaults to it, under /var/lib/valka/data.
docker run -d --name valka --restart unless-stopped \
-p 8989:8989 -p 50051:50051 \
-v valka-data:/var/lib/valka \
ghcr.io/iwhitebird/valka:latestOr run the binary directly; it uses ./data by default:
valka-server # or: valka-server /etc/valka/valka.tomlThat directory is the bucket: the WAL, snapshots and task logs live there. Back it up like a
database, and never point two processes at it. The local backend is single-node only; a
cluster needs an S3-compatible bucket.
With an S3-compatible bucket
Point the node at a bucket with the s3 backend:
docker run -d --name valka --restart unless-stopped \
-p 8989:8989 -p 50051:50051 \
-e VALKA_NODE_ID=valka-1 \
-e VALKA_STORAGE__BACKEND=s3 \
-e VALKA_STORAGE__BUCKET=my-valka-bucket \
-e VALKA_STORAGE__REGION=us-east-1 \
ghcr.io/iwhitebird/valka:latestCredentials come from VALKA_STORAGE__ACCESS_KEY_ID and VALKA_STORAGE__SECRET_ACCESS_KEY, or,
when those are unset, from the AWS default chain (environment, profile, instance metadata, IRSA).
For MinIO and other S3-compatible stores, also set VALKA_STORAGE__ENDPOINT (and
VALKA_STORAGE__ALLOW_HTTP=true for a plain-HTTP endpoint in development).
Set a fixed VALKA_NODE_ID. It defaults to the host name, which changes with every new
container unless you set a hostname. A restarted node with the same id fences its previous run,
splits what that run left in the log and reclaims its shards at once.
With Docker Compose
The repository's docker-compose.yml starts an S3-compatible store and one node built from
source:
git clone https://github.com/IWhitebird/Valka.git && cd Valka
docker compose up -d| Service | Port | Description |
|---|---|---|
valka-server | 8989, 50051 | REST, dashboard, gRPC. Node id valka-1 |
minio | 9000, 9001 | S3 API and web console (minioadmin / minioadmin) |
minio-init | - | Creates the valka and valka-test buckets, then exits |
The store is pgsty/silo, an S3-compatible MinIO fork (the minio/minio images are no longer
published on Docker Hub). In production, point [storage] at a real bucket and drop the store
services. deploy/docker/docker-compose.yml is a variant of the same setup whose bucket,
endpoint and credentials can be overridden with environment variables (S3_BUCKET,
S3_ENDPOINT, S3_ACCESS_KEY_ID, ...).
A three-node cluster with Docker Compose
docker-compose.cluster.yml runs three nodes on one S3-compatible store:
docker compose -f docker-compose.cluster.yml up -d --build| Node | REST API and dashboard | gRPC |
|---|---|---|
| valka-1 | http://localhost:8981 | localhost:50061 |
| valka-2 | http://localhost:8982 | localhost:50062 |
| valka-3 | http://localhost:8983 | localhost:50063 |
Every node gets the same storage settings, VALKA_CLUSTER__ENABLED=true, the same cluster token
and an admin token, plus its own VALKA_NODE_ID and advertised addresses. Change both tokens
before using it beyond a laptop. The build is limited to 4 parallel compile jobs to keep a laptop
usable; it takes a while the first time.
See Clustering to watch a takeover on it.
VMs and bare metal
Run one valka-server per machine, all pointing at the same bucket. Every node needs:
- the same
[storage]settings (bucket, prefix, endpoint, region); VALKA_CLUSTER__ENABLED=trueand the sameVALKA_CLUSTER__TOKEN;- a distinct, stable
VALKA_NODE_ID; - advertised addresses the other nodes and clients can reach.
VALKA_NODE_ID=valka-1
VALKA_STORAGE__BACKEND=s3
VALKA_STORAGE__BUCKET=my-valka-bucket
VALKA_STORAGE__REGION=eu-west-1
VALKA_CLUSTER__ENABLED=true
VALKA_CLUSTER__TOKEN=<a long random secret, the same on every node>
VALKA_CLUSTER__INTERNAL_ADDR=0.0.0.0:50052
VALKA_CLUSTER__ADVERTISE_INTERNAL_ADDR=10.0.0.1:50052
VALKA_CLUSTER__ADVERTISE_GRPC_ADDR=10.0.0.1:50051
VALKA_CLUSTER__ADVERTISE_HTTP_ADDR=10.0.0.1:8989
VALKA_API_KEYS=<keys for producers and workers, comma-separated>
VALKA_ADMIN_TOKEN=<another secret, only if you want the admin API>Nodes 2 and 3 differ only in VALKA_NODE_ID and their addresses. Open 50052 between the nodes
and nowhere else. For mTLS on the internal port, see Clustering.
Give workers and clients every node's address (all four SDKs and the CLI take a list), or a load
balancer or DNS name in front of all of them, so a worker that loses its node reconnects to another
one and a create that hits a dying node is retried on the next. Clients send one of the API keys
(VALKA_API_KEY in the SDKs and the CLI).
No systemd units ship yet. Any supervisor works, as long as it restarts the process when it exits and gives it time to stop (see Graceful shutdown).
Bucket requirements
The protocol relies on the bucket's consistency, so a cluster needs a store with:
- conditional writes: create-if-absent PUT and
If-MatchPUT; - strong read-after-write and strong LIST consistency;
- stable ETags.
AWS S3 and recent MinIO provide these. A clustered node runs a storage self-test at start: it creates a probe object twice (the second create must fail), CASes with a wrong ETag (must fail) and with the right one (must succeed), and checks that LIST shows the probe at once. If any check fails, the node refuses to start.
Use one bucket, or one prefix inside a bucket, per cluster. The first node writes
cluster.json with the shard count and the storage format; every later node must agree with it.
Capacity
A node holds the live tasks of the shards it owns in RAM, plus finished tasks for
completed_retention_secs (24 h by default); their history stays in the bucket. Adding a node
adds capacity: placement moves about 1/N of the shards to it after the join delay. Useful nodes
top out around num_shards / 8, so choose the shard count of a new bucket with growth in mind.
Kubernetes
The Helm chart in deploy/helm predates the WAL architecture and is being reworked. Don't use
it with this version. Until the new chart lands, run the container yourself as a StatefulSet-like
deployment, or use Docker Compose or VMs:
- one pod per node, with a stable node id: in a StatefulSet the pod name is the host name, so the default node id is already stable;
- a headless service, with
VALKA_CLUSTER__ADVERTISE_INTERNAL_ADDRset to each pod's stable DNS name and port50052; - the same storage settings and cluster token for every pod (the token from a Secret);
- no volume needed with the
s3backend; - a termination grace period long enough for the node to hand its shards over on
SIGTERM; - liveness and readiness probes on
GET /healthz(port 8989); - port
50052reachable only from the other pods.
Chaos test
scripts/chaos.sh breaks the three-node compose cluster while a load generator runs:
scripts/chaos.sh # 5 minutes of chaos
DURATION=1800 scripts/chaos.sh # a 30-minute soak
NO_BUILD=1 scripts/chaos.sh # reuse the valka:local imageThe load generator (examples/rs/chaos.rs) creates tasks through every node (20 per second by
default, RATE), runs a worker on every node and records every acknowledged create. Meanwhile
the script picks a random node every 10–30 s and either kill -9s it, freezes it (SIGSTOP,
like a long GC pause or a VM freeze) or restarts it with SIGTERM. At the end every node comes
back, and the checker verifies that every acknowledged task exists and finished, and that no
idempotency key produced two tasks. It needs Docker Compose and cargo, writes its logs to
target/chaos, and exits 0 when every check passes.
Monitoring and alerts
Health checks
curl http://localhost:8989/healthz # "ok" (200), or 503 "wal writer poisoned"
curl http://localhost:8989/metrics # Prometheus text format/healthz answers 503 only when the node's WAL writer is poisoned, and such a node exits right
after. It does not report a self-fenced node (one that can't renew its lease); watch the
valka_node_serving metric for that. The Docker image has a built-in HEALTHCHECK on
/healthz.
livenessProbe:
httpGet:
path: /healthz
port: 8989
initialDelaySeconds: 15
periodSeconds: 10
readinessProbe:
httpGet:
path: /healthz
port: 8989
periodSeconds: 5For a cluster-wide view, GET /api/v1/cluster on any node returns every node, the number of
unowned shards and an overall health.status (ok, degraded or critical).
Metrics
Every node exports /metrics on its HTTP port. The cluster metrics:
| Metric | Meaning |
|---|---|
valka_node_serving | 1 while the node's lease is current, 0 when it self-fenced |
valka_lease_age_seconds | Time since the last successful lease renewal |
valka_shards_owned | Shards this node serves |
valka_cluster_members | Nodes this node sees |
valka_node_fenced_total | Times this node found its lease taken |
valka_tombstones_total | Nodes declared dead |
valka_splits_total, valka_split_failures_total, valka_split_duration_seconds | Log splits |
valka_shard_claims_total, valka_shard_releases_total | Shard moves |
valka_forwarded_commands_total | Requests forwarded to other nodes, by result |
valka_route_redirects_total | NOT_OWNER redirects followed |
valka_reattach_total | Worker runs reattached after a reconnect |
valka_wal_segments_total, valka_wal_bytes_total | WAL segments and bytes written |
Task metrics include valka_tasks_created_total, valka_tasks_completed_total,
valka_tasks_failed_total, valka_tasks_retried_total, valka_tasks_dead_lettered_total,
valka_pending_tasks, valka_active_workers and valka_task_duration_ms.
Alert rules
deploy/prometheus/alerts.yml has ready-made Prometheus rules. Each one names a section of the
runbooks, which say what
to do when it fires.
| Alert | Fires when | Runbook |
|---|---|---|
ValkaNodeNotServing (critical) | A node self-fenced for 30 s | Node not serving |
ValkaLeaseStale | Lease renewals take over 6 s for 1 min | Bucket slow or down |
ValkaNodeFenced | A node found its lease taken | Node fenced |
ValkaTakeover | A node was declared dead and taken over | Node lost |
ValkaSplitFailing (critical) | A dead node's log split keeps failing | Stuck split |
ValkaUnownedShards (critical) | Fewer shards served than exist, for 2 min | Unowned shards |
ValkaClaimInvariant (critical) | A claim found the bucket inconsistent with the protocol | Claim invariant |
ValkaForwardingErrors | Nodes can't reach each other's internal listener | Nodes cannot reach each other |
ValkaUnownedShards compares against 4096. Change the number if your cluster was created with
another shard count.
Graceful shutdown
Stopping a server (SIGTERM or Ctrl-C):
- Every connected worker receives
ServerShutdownand its stream is closed, so the server never waits on open streams. - Workers keep running their in-flight tasks and reconnect with backoff, to another node or to the restarted one. Results that finish in the meantime are kept and delivered after the reconnect.
- If live peers exist, the node drains: it marks its lease draining, releases every shard with a final snapshot, marks its lease dead and exits. The peers claim the shards with nothing to replay.
- If it is alone, it keeps its shards: it flushes the WAL, snapshots changed shards and exits. The restart reclaims them with little to replay.
Give the process time to finish. Docker's default stop timeout is 10 s; the chaos script uses
docker compose restart -t 30.
Rolling restart. Restart one node at a time, and wait until GET /api/v1/cluster shows no
unowned shards before the next one. Each SIGTERM hands that node's shards to the others.
Drain without a restart. POST /api/v1/cluster/nodes/{node}/drain (admin API) makes a node
hand everything over and exit.
Stopping a worker:
- The worker sends
GracefulShutdown: the server stops sending it tasks but keeps the stream open. - In-flight handlers finish and their results are acknowledged (the worker waits up to 5 s for the acks).
- The worker closes the stream and exits.
When the bucket fails
- Writes are acknowledged only after their WAL segment is in the bucket, so a slow bucket means
slower acknowledgements and, eventually,
UNAVAILABLE. Acknowledged work is never lost. - A node that can't renew its lease for 7 s stops serving until it can. If it stays cut off for 10 s, another node takes its shards over.
- If WAL PUTs keep failing after their retries (
put_retries, default 8), the node fails the pending writes withUNAVAILABLEand exits with a non-zero status. Its supervisor restarts it, and it rebuilds its state from the bucket. - If the bucket is down for everyone, every node stops serving and nobody can declare anybody dead. The cluster recovers by itself when the bucket returns.
Upgrades
Restart nodes one at a time onto the new build, as in a rolling restart. Each lease advertises
the node's protocol version, and the cluster's effective protocol is the lowest among live nodes
(effective_protocol in GET /api/v1/cluster/placement). A node refuses to join a cluster whose
live nodes already speak a newer protocol.
Configuration reference
Configuration is layered: built-in defaults, then a TOML file, then VALKA_* environment
variables (highest wins). Nested keys use a double underscore: [storage] bucket is
VALKA_STORAGE__BUCKET. Pass the config file path as the first argument:
valka-server /etc/valka/valka.toml.
Environment variables
| Variable | Default | Description |
|---|---|---|
VALKA_NODE_ID | host name | Stable node id (1–64 characters from A-Z a-z 0-9 _ -) |
VALKA_ADMIN_TOKEN | - | Enables the admin API |
VALKA_API_KEYS | - | API keys for the public API, comma-separated. Unset: the API is open (the server warns) |
VALKA_LIMITS__MAX_TASKS_IN_RAM | 1000000 | Tasks a node holds in RAM; above it creates get 429 |
VALKA_LIMITS__MAX_PAYLOAD_BYTES | 1048576 | Largest task input or metadata |
VALKA_GRPC_ADDR | 0.0.0.0:50051 | gRPC listen address |
VALKA_HTTP_ADDR | 0.0.0.0:8989 | REST, dashboard and metrics listen address |
VALKA_WEB_DIR | web/dist | Built dashboard files (the image sets /usr/share/valka/web) |
VALKA_STORAGE__BACKEND | local | s3, local (a directory) or memory (tests only) |
VALKA_STORAGE__PATH | ./data | Directory for the local backend |
VALKA_STORAGE__BUCKET | valka | Bucket for the s3 backend |
VALKA_STORAGE__PREFIX | empty | Key prefix inside the bucket, e.g. valka/prod |
VALKA_STORAGE__ENDPOINT | - | Custom S3 endpoint (MinIO, R2, ...). Omit for AWS |
VALKA_STORAGE__REGION | - | Bucket region |
VALKA_STORAGE__ACCESS_KEY_ID, VALKA_STORAGE__SECRET_ACCESS_KEY | - | Credentials. Unset: the AWS default chain |
VALKA_STORAGE__ALLOW_HTTP | false | Allow plain-HTTP endpoints (development only) |
VALKA_WAL__FLUSH_INTERVAL_MS | 50 | Group-commit window: acknowledgement latency vs. PUT count |
VALKA_WAL__MAX_BATCH_BYTES | 4194304 | Flush early at this many buffered bytes |
VALKA_WAL__MAX_INFLIGHT_SEGMENTS | 4 | Segment PUTs in flight at once |
VALKA_WAL__PUT_RETRIES | 8 | PUT retries before writes fail and the node exits |
VALKA_WAL__SNAPSHOT_INTERVAL_SECS | 60 | Snapshot changed shards at least this often |
VALKA_WAL__SNAPSHOT_AFTER_RECORDS | 50000 | ...or once a shard has this many new records |
VALKA_WAL__LOG_BUDGET_BYTES | 67108864 | ...or once this much log is uncovered. Bounds restart and takeover time |
VALKA_WAL__SNAPSHOTS_TO_KEEP | 2 | Snapshots kept per shard |
VALKA_WAL__COMPLETED_RETENTION_SECS | 86400 | Finished tasks leave RAM after this (their history stays in the bucket) |
VALKA_CLUSTER__* | Cluster settings: see Clustering | |
VALKA_SCHEDULER__TIMER_TICK_MS | 100 | Timer resolution for lease expiry, retries and delayed tasks |
VALKA_SCHEDULER__LEASE_GRACE_SECS | 30 | Added to a task's timeout for its first lease |
VALKA_SCHEDULER__HEARTBEAT_LEASE_SECS | 60 | Each heartbeat keeps the lease at least this far ahead |
VALKA_SCHEDULER__RECOVERY_GRACE_SECS | 60 | Lease given to recovered RUNNING tasks until their worker reconnects |
VALKA_SCHEDULER__RETRY_BASE_DELAY_SECS | 1 | Retry backoff base |
VALKA_SCHEDULER__RETRY_MAX_DELAY_SECS | 3600 | Retry backoff cap |
RUST_LOG | valka=info,tower_http=info | Log filter |
Full configuration file
Every setting with its default and a comment, from deploy/valka.example.toml:
# Valka Server Configuration Reference
#
# Config layering: defaults -> this file -> VALKA_* env vars
# Nested env vars use double underscore: VALKA_STORAGE__BUCKET=valka
# Pass config path as first CLI argument: valka-server /etc/valka/valka.toml
# Unique, stable node identifier. Defaults to the host name if empty.
# A restarted node with the same id fences its previous run, splits what that run left in
# wal/<node_id>/ and reclaims its shards. Every node of a cluster needs a distinct id.
node_id = ""
# Bearer token for the admin API (drain a node, move a shard, set queue partitions,
# audit log). Unset = admin API disabled. Env: VALKA_ADMIN_TOKEN.
# admin_token = ""
# API keys for the public REST and gRPC APIs, comma-separated (env: VALKA_API_KEYS).
# Clients send one as "Authorization: Bearer <key>"; the SDKs and the CLI read
# VALKA_API_KEY. Unset = the public API is OPEN to anyone who can reach its ports (the
# server logs a warning). /healthz and /metrics stay open.
# api_keys = "key-for-producers,key-for-workers"
# gRPC listen address (workers and API clients)
grpc_addr = "0.0.0.0:50051"
# HTTP/REST + WebUI listen address
http_addr = "0.0.0.0:8989"
# Path to built WebUI static files
web_dir = "/usr/share/valka/web"
# --- Storage (the only durable state) --------------------------------------
[storage]
# "s3" any S3-compatible bucket (AWS S3, MinIO, R2, ...). Needs conditional writes.
# "local" a directory (dev / single machine). CAS is emulated in-process.
# "memory" ephemeral, tests and demos only.
backend = "s3"
# local backend: directory
path = "./data"
# s3 backend
bucket = "valka"
# Optional key prefix inside the bucket, e.g. "valka/prod"
prefix = ""
# Custom endpoint for S3-compatible stores (MinIO: "http://minio:9000"). Omit for AWS.
# endpoint = "http://minio:9000"
# region = "us-east-1"
# Credentials. Omit to use the AWS default chain (env vars, profile, IMDS / IRSA).
# access_key_id = ""
# secret_access_key = ""
# Allow plain HTTP endpoints (MinIO in dev only)
allow_http = false
# --- Write-ahead log -------------------------------------------------------
[wal]
# Group-commit window. Nothing is acknowledged before its segment is in the bucket.
# Lower = lower enqueue latency, more PUT requests. 50-100 ms is the sweet spot on S3.
flush_interval_ms = 50
# Flush early once the buffer reaches this many bytes.
max_batch_bytes = 4194304
# Segment PUTs in flight concurrently.
max_inflight_segments = 4
# PUT retries before acks fail with UNAVAILABLE (the node then exits and restarts).
put_retries = 8
# Snapshot dirty shards at least this often; segments below every snapshot are deleted.
snapshot_interval_secs = 60
# ...or as soon as a shard has this many unsnapshotted records.
snapshot_after_records = 50000
# ...or, for every dirty shard, once this many WAL bytes were written since the last
# full round. Bounds restart replay time. 0 disables.
log_budget_bytes = 67108864
# Snapshots kept per shard.
snapshots_to_keep = 2
# Terminal tasks (COMPLETED/FAILED/CANCELLED/DEAD_LETTER) leave RAM after this.
# Their history stays in the bucket.
completed_retention_secs = 86400
# --- Cluster ---------------------------------------------------------------
# One node is a cluster of one: the same lease / ownership protocol always runs.
# See docs/ARCHITECTURE.md §8 and docs/wal/PHASE2.md.
[cluster]
# Allow other nodes on this bucket. Needs the s3 backend (local/memory CAS only works
# inside one process) and a token.
enabled = false
# Shared secret every node-to-node call carries. Required when enabled.
# token = ""
# Shard count of a NEW bucket (written to cluster.json on first start; fixed afterwards).
# Power of two, 16..=65536. 0 = whatever the bucket says, or 4096 for a new bucket.
num_shards = 0
# Node-to-node listener. Never expose it publicly.
internal_addr = "0.0.0.0:50052"
# Addresses other nodes / clients use to reach this node (default: the listen addresses).
# advertise_internal_addr = "10.0.0.1:50052"
# advertise_grpc_addr = "10.0.0.1:50051"
# advertise_http_addr = "10.0.0.1:8989"
# Optional mTLS for internal traffic (PEM files). The token is still required.
# tls_cert = "/etc/valka/tls/node.crt"
# tls_key = "/etc/valka/tls/node.key"
# tls_ca = "/etc/valka/tls/ca.crt"
# Lease renewal period, and how long an unchanged lease takes to be declared dead
# (by the observer's own clock). A node that has not renewed for
# lease_timeout - lease_safety_margin stops serving on its own.
lease_interval_ms = 2000
lease_timeout_ms = 10000
lease_safety_margin_ms = 3000
# How often nodes/ is listed (membership).
membership_interval_ms = 3000
# A new node takes shards after this (all nodes at once on a cold start).
join_delay_ms = 15000
# Placement round period, claims in flight, shards handed over per round.
placement_interval_ms = 3000
claim_concurrency = 64
handover_per_tick = 64
# A shard stays where it is for at least this long after a move.
min_residency_secs = 300
# Another node may redo a dead node's log split after this.
split_timeout_ms = 30000
# After claiming shards, recovered PENDING tasks wait this long so reconnecting workers
# can reattach their runs first.
reconcile_grace_ms = 5000
# Garbage collector cadence (dead logs, old snapshots, split markers).
gc_interval_secs = 300
# Refuse to start while another live process holds this node id.
startup_lease_check = true
# --- Limits ------------------------------------------------------------------
[limits]
# Tasks a node may hold in RAM (live, plus finished ones within completed_retention_secs).
# Above it, creates are refused with RESOURCE_EXHAUSTED / HTTP 429 until the backlog
# drains; near it, finished tasks leave RAM early (their history stays in the bucket).
# 0 disables the limit.
max_tasks_in_ram = 1000000
# Largest task input or metadata, in bytes of JSON. Store big data elsewhere and pass a
# reference.
max_payload_bytes = 1048576
# --- Matching / Task Routing -----------------------------------------------
[matching]
# In-RAM worker-routing partitions per queue on one node (unrelated to storage shards
# and to queue partitions, which live in the bucket: queues/{hh}/{queue}).
num_partitions = 4
# Internal tree branching factor (rarely needs tuning)
branching_factor = 3
# Max tasks buffered per partition before back-pressure
max_buffer_per_partition = 1000
# How often the feeder tops up matching buffers from the pending index (ms)
feeder_interval_ms = 20
# Max tasks moved per queue per feeder tick
feeder_batch_size = 200
# --- Scheduler -------------------------------------------------------------
[scheduler]
# Timer resolution for lease expiry, retry promotion and delayed tasks (ms)
timer_tick_ms = 100
# Extra time on top of a task's timeout before its first lease expires (seconds)
lease_grace_secs = 30
# Lease granted to RUNNING tasks recovered from the WAL until their worker reconnects
recovery_grace_secs = 60
# Lease granted by each heartbeat: the worker must heartbeat again within this window
heartbeat_lease_secs = 60
# Exponential backoff base for retries (seconds)
retry_base_delay_secs = 1
# Maximum retry delay cap (seconds)
retry_max_delay_secs = 3600
# --- Log Ingester ----------------------------------------------------------
[log_ingester]
# Max log lines buffered before a chunk is written to the bucket
batch_size = 100
# Max time to buffer logs before flushing (ms)
flush_interval_ms = 500