Workers & Queues
Settings → Workers shows every background queue serving the workspace: its state, its waiting backlog, how many batches it ran, which worker processes report on it, and a per-queue pause switch. Above the table sits the Pause workspace card — pausing the whole workspace stops all of its workers at once (see Pause & Resume).
Reading the table
- State.
running(a batch is executing),waiting_slot(queued for a concurrency slot),failed(the last batch errored — details under the queue name),idle, orstale, shown as No heartbeat. - Backlog. Jobs waiting to be fetched, read from the queue itself — an idle-looking queue with a backlog still has work nobody picked up.
- Runs. Completed batches, with failures called out separately.
- Workers. The processes currently reporting on the queue, truncated with
the full list in the tooltip.
hostname:pididentifies the pod and process.
No heartbeat
Every worker process reports every 5 seconds; a row silent for more than 30 seconds is downgraded to stale rather than believed, so an OOM-killed pod cannot leave a queue looking busy forever. Rows silent for more than 5 minutes are deleted outright (clean shutdowns already remove their own row), so corpse entries cannot pin a queue at stale or pile dead ids into the Workers column.
Two cases where stale is correct, not a bug: a queue with a backlog but no live worker (nothing beats for it — start or resume the workers), and a paused workspace (its workers are stopped, so there is no heartbeat until resume).
Pausing a queue
The per-queue switch stops fetching: queued jobs wait safely and run after resume instead of burning retries, and a batch fetched just as the pause landed waits for resume rather than failing. Jobs already running cannot be cancelled — pause and let them drain, or restart the worker. Pausing is database-backed, so it applies to every worker replica.
Queues held by a feature
Turning embeddings or duplicate detection off in Settings → Cleanup holds that feature’s queues paused:
| Feature | Queues |
|---|---|
| Duplicate detection | correlation.scan |
| Embeddings | semantic-embeddings-<space>, semantic-recalibrate-<space> |
A held queue shows Held · <feature> off instead of Paused, and offers
a link to the switch instead of Resume — resuming it here would run work
the workspace turned off, so the API refuses with 409. Turning the feature
back on releases the hold. A queue you had also paused by hand before stays
paused after that; one you tried to resume while it was held resumes with the
feature.
autopilot.handoff carries a finished scan to the AI harness while duplicate
detection is off (normally that hand-off rides on correlation.scan, after the
duplicate check). It is never held.
Purging a held queue’s backlog is allowed, and rarely useful: while a feature is off nothing new is queued for it.
Purging a backlog
A queue with waiting jobs offers Purge, behind a confirmation that names
the count. Purging deletes the waiting (created) and scheduled-retry jobs
of that one queue:
- Dropped jobs are gone, not re-queued. Periodic schedules recover on their next fire; one-off work (a manually scheduled recompute) needs a manual re-trigger.
- Jobs waiting to retry a failure go with the backlog: a failure that would have recovered on retry stays failed (its error row remains visible).
- Running jobs finish normally. Finished history and the cron schedules themselves are never touched — for finished-job history see Storage & Cleanup instead.
- Housekeeping queues (
__pgboss__*) refuse with a 400; purging is allowed while the workspace is paused (nothing refills the queue until resume) and needsworkspace.settings.
Why there is no “run now” button
Queue jobs carry queue-specific payloads (which assets, which inquiry, which source), so a generic trigger would fire handlers with empty input. Manual starts live where the work is defined instead: scan now on a source, autopilot trigger, embedding reindex, correlation recompute. The queues tab stays a visibility and wind-down surface: watch, pause, purge.