Skip to Content
SettingsWorkers & Queues

Workers & Queues

Settings → Workers shows every background queue serving the workspace: its state, its waiting backlog, how many batches it ran, which worker processes report on it, and a per-queue pause switch. Above the table sits the Pause workspace card — pausing the whole workspace stops all of its workers at once (see Pause & Resume).

Reading the table

  • State. running (a batch is executing), waiting_slot (queued for a concurrency slot), failed (the last batch errored — details under the queue name), idle, or stale, shown as No heartbeat.
  • Backlog. Jobs waiting to be fetched, read from the queue itself — an idle-looking queue with a backlog still has work nobody picked up.
  • Runs. Completed batches, with failures called out separately.
  • Workers. The processes currently reporting on the queue, truncated with the full list in the tooltip. hostname:pid identifies the pod and process.

No heartbeat

Every worker process reports every 5 seconds; a row silent for more than 30 seconds is downgraded to stale rather than believed, so an OOM-killed pod cannot leave a queue looking busy forever. Rows silent for more than 5 minutes are deleted outright (clean shutdowns already remove their own row), so corpse entries cannot pin a queue at stale or pile dead ids into the Workers column.

Two cases where stale is correct, not a bug: a queue with a backlog but no live worker (nothing beats for it — start or resume the workers), and a paused workspace (its workers are stopped, so there is no heartbeat until resume).

Pausing a queue

The per-queue switch stops fetching: queued jobs wait safely and run after resume instead of burning retries, and a batch fetched just as the pause landed waits for resume rather than failing. Jobs already running cannot be cancelled — pause and let them drain, or restart the worker. Pausing is database-backed, so it applies to every worker replica.

Queues held by a feature

Turning embeddings or duplicate detection off in Settings → Cleanup holds that feature’s queues paused:

FeatureQueues
Duplicate detectioncorrelation.scan
Embeddingssemantic-embeddings-<space>, semantic-recalibrate-<space>

A held queue shows Held · <feature> off instead of Paused, and offers a link to the switch instead of Resume — resuming it here would run work the workspace turned off, so the API refuses with 409. Turning the feature back on releases the hold. A queue you had also paused by hand before stays paused after that; one you tried to resume while it was held resumes with the feature.

autopilot.handoff carries a finished scan to the AI harness while duplicate detection is off (normally that hand-off rides on correlation.scan, after the duplicate check). It is never held.

Purging a held queue’s backlog is allowed, and rarely useful: while a feature is off nothing new is queued for it.

Purging a backlog

A queue with waiting jobs offers Purge, behind a confirmation that names the count. Purging deletes the waiting (created) and scheduled-retry jobs of that one queue:

  • Dropped jobs are gone, not re-queued. Periodic schedules recover on their next fire; one-off work (a manually scheduled recompute) needs a manual re-trigger.
  • Jobs waiting to retry a failure go with the backlog: a failure that would have recovered on retry stays failed (its error row remains visible).
  • Running jobs finish normally. Finished history and the cron schedules themselves are never touched — for finished-job history see Storage & Cleanup instead.
  • Housekeeping queues (__pgboss__*) refuse with a 400; purging is allowed while the workspace is paused (nothing refills the queue until resume) and needs workspace.settings.

Why there is no “run now” button

Queue jobs carry queue-specific payloads (which assets, which inquiry, which source), so a generic trigger would fire handlers with empty input. Manual starts live where the work is defined instead: scan now on a source, autopilot trigger, embedding reindex, correlation recompute. The queues tab stays a visibility and wind-down surface: watch, pause, purge.

Last updated on