Storage & Cleanup

Settings → Cleanup has two parts:

• Features — on/off switches for the two background engines that build most of a workspace’s derived data: embeddings and duplicate detection. Turning one off stops its work everywhere, pauses its workers, and asks whether to keep or delete what it built.
• Datasets — every storage dataset of the workspace with live row estimates (~, from the planner) and byte sizes (indexes included), plus a workspace total, and a safe way to wipe the ones that can be rebuilt or are only history.

Neither part ever touches the investigation itself: sources, findings, assets, cases, inquiries and the glossary.

Feature switches

Both are on in a new workspace. (Embeddings follow the deployment’s default until a workspace chooses; a deployment that ships them off shows Off by default.) Each row shows the feature’s state, when it last changed, how much data it owns right now, and — while off — which workers it holds paused.

How each feature works is configured elsewhere: the embedding model, provider and throughput live in Harness AI → Embedding, duplicate matching in Duplicates → Tuning. Both link back here for the switch, so there is exactly one place to turn either on or off.

Turning a feature off

The switch opens a dialog that lists what stops while the feature is off, then asks how to turn it off:

• Turn off, keep data — a pause. Everything the feature built stays as it is, frozen, and turning it back on picks up where it stopped.
• Turn off and delete data — frees the storage now, in the background, with live progress. Turning it back on rebuilds from scratch.

Either way:

• Work stops in every process within seconds — API pods and worker pods alike, on every replica.
• Its workers are held. The feature’s queues are paused and show as Held · <feature> off in Settings → Workers. They cannot be resumed there — the API answers 409 — because resuming them would run work the workspace switched off. Turning the feature back on releases them.
• The investigation is never touched, and neither is anything a person decided: review decisions, cases, inquiries.

Embeddings

Why a pause keeps saving text chunks: chunks are the document text the vectors are built from, and a scan never sends them twice — the scan cache marks an unchanged file as done. Skipping them while paused would leave those files permanently without semantic coverage. Deleting the data is the explicit choice to stop collecting them.

Agents that wait for evidence ranking are not held back while embeddings are off: a paused backlog would never drain, so the harness stops waiting on it and works from the evidence as it stands.

Duplicate detection

Scans still reach the AI harness while duplicate detection is off. A finished scan normally hands its source to the autopilot after the duplicate check, on the same queue; with that queue held, the hand-off travels on its own queue (autopilot.handoff) instead, and scans that were already waiting are moved there when the switch goes off.

A recompute that is running when the switch goes off stops at its next page of assets — it does not finish writing another few million pairs first — and is recorded in the harness log as skipped, not failed.

Turning a feature back on

The switch opens a dialog that says what catching up costs before it starts:

• Embeddings resume into the same vector space: a catch-up pass embeds everything scanned while they were off, and rankings update as it goes. After delete, everything is embedded from scratch instead.
• Duplicate detection runs one full recompute over every asset. On a large corpus that is about an hour of worker time; Duplicate review fills in as it finishes. It is one recompute, not a backlog — the scans, edits and tuning changes made while it was off were deliberately not queued one by one.

Deleting what a paused feature kept

A feature that is off with its data kept offers Delete kept data. It stays off; the data is deleted in the background, exactly as if it had been turned off with delete in the first place (for embeddings, scans then also stop saving text chunks).

Clean in the dataset table below does the same for a feature that is off. For a feature that is on, Clean is a one-off wipe and the data starts growing back with the next scan or recompute — the confirmation says so, and suggests turning the feature off instead if the goal is to keep it empty.

Where the switches show up

Every notice links back to this tab.

From the API or MCP

GET /<workspace>/maintenance/features
PUT /<workspace>/maintenance/features/duplicates   {"enabled": false}
PUT /<workspace>/maintenance/features/embeddings   {"enabled": false, "deleteData": true}
PUT /<workspace>/maintenance/features/embeddings   {"enabled": true}

The PUT answers with the new state, the id of the cleanup run when data is being deleted (poll GET /maintenance/cleanup/runs/:runId), and whether a catch-up recompute was queued. It is allowed while the workspace is paused and blocked in demo mode. Turning a feature on or off while its data is being deleted answers 409.

Work refused because a feature is off answers with a message naming the switch: a manual duplicate recompute or review rebuild (409), an embedding reindex, recalibration or rebuild (409), a similar-findings or semantic query (503 — search callers fall back to exact text). The embedding settings endpoint no longer accepts enabled; the MCP get_correlation_config tool reports enabled, and saving tuning while duplicate detection is off says that nothing recomputes until it is turned back on.

Datasets

The dataset tables list every storage dataset of the workspace. Two groups, two rules:

• Measured, never deleted. Sources, findings, assets, cases, inquiries and glossary carry a Kept badge and have no action — the API has no cleanup path for them at all, so a misclick cannot destroy the investigation.
• Cleanable. Scan history, duplicate artefacts, embeddings, harness history, finished queue jobs, derived stats/graphs and finished transfers. Each row explains what goes, what stays, and what rebuilds itself.

Every cleanup asks for confirmation first, then runs in the background: the request returns immediately (202 + a run id) while the row shows live progress — rows removed so far, the table currently draining, and a bar against the starting estimate. Big wipes take minutes; you can leave the tab open and watch, or come back later — the table reloads with the final result (removed per table, live work left alone).

Pause first when workers are active. Cleanup never touches running scans or queued jobs, but live workers keep writing while the wipe drains — an embedding pipeline can re-insert thousands of rows into a table you are emptying, and the tab will look “still full” afterwards. When the overview reports pending/running scans, pause the workspace (Settings → Pause), run the cleanup, then resume. Paused means frozen: no writer can race the wipe. For embeddings and duplicates, turning the feature off does the same for that one engine.

One table, two datasets

The asset graph (edges) holds two very different things: lineage — how assets derive from each other, which only a source rescan can rebuild — and the scored duplicate links duplicate detection writes, which the next duplicate check rebuilds. The overview splits the table’s size between the Assets row (lineage) and the Duplicate artefacts row (scored links) by the planner’s statistics, and the duplicates cleanup removes only the scored links. On the incident that motivated this, the scored links were 17 GB of a 42 GB database, all of it previously counted as protected assets.

What each cleanup does

Side effects

Case and inquiry rows are never touched by any cleanup: no case or inquiry table references cleanable storage, so triage work always survives whatever you wipe. What each wipe does change, besides freeing space:

• Scan history (destructive, no rebuild). Finished runs vanish from the scans list and their detail pages stop resolving; pending and running scans are skipped, schedules and future scans are unaffected. Assets lose their “last seen by” run link (the assets themselves stay); notifications keep their text but lose the run link; findings keep the deleted run id as plain filterable text.
• Duplicate artefacts (rebuilds on the next duplicate check). Duplicate review, “where else found” and the graph’s similarity links go empty until the check runs again — automatically as part of scanning, or via a manual full recompute. Human verdicts and review batches are kept, so triage decisions survive; lineage links are kept, so the lineage view is unchanged. The wipe waits for a duplicate check that is already running rather than racing it.
• Embeddings. Importance scores read 0 and semantic search, similar findings and near-duplicate text groups return nothing until re-embedding finishes. While embeddings are on, findings are re-embedded automatically; document text chunks only come back when sources are rescanned with Full rescan (in the scan wizard), because the scan cache skips files that did not change. Embedding settings stay.
• AI harness history (destructive, no rebuild). Agent runs, including dreams, vanish with their logs and decisions, and the harness history views go empty. Learned memory, goals, briefs, configs and undo history stay, and nothing outside the history views reads past runs — future autopilot cycles behave exactly as before.
• Finished queue jobs (history only). The worker-queues history loses finished entries; pending, scheduled and running jobs keep running and the cron schedules themselves are never touched, so scheduling continues unchanged.
• Finding statistics (rebuilds on refresh). Dashboards and trends read the rollups, so they show zeros until the next refresh (demand-driven, plus a manual refresh). The first refresh after a wipe is a full recompute, not an incremental one — expect a heavier worker pass.
• Source graph (rebuilds via the graph worker). The source graph view goes empty until the worker rebuilds it; same full-recompute caveat as the stats above.
• Finished transfers (history only). Completed import/export entries disappear from the transfer history; staged and running transfers continue untouched.

How it stays safe on huge tables

Workspaces in production hold tens of millions of rows (a busy one can keep 8M+ embedding chunks at 16 GB). Cleanup is built for that scale:

• TRUNCATE first, chunked DELETE as fallback. Whole-dataset wipes use a single instant TRUNCATE whenever no outside table references the dataset (checked live against the catalog, with a bounded 15 s lock wait so a busy writer cannot wedge the request). If anything references it — or the lock wait expires — the wipe drains 2,000 rows at a time via ctid positionals: short statements, row locks only, no long transaction, no table lock.
• Shared tables are filtered, never truncated. The scored links in the asset graph are removed one of two ways. When they are most of the table and the lineage kept is small, the table is rewritten: lineage is copied aside, the table truncated and the lineage copied back, in one transaction that holds the table so no concurrent write is lost — the only way to give the disk back at once. Otherwise the rows are deleted by physical block range, 64 MB of the table per statement: sequential reads, no index needed.
• Embeddings go completely. Each vector space’s search index and its two worker queues are dropped with it (both are named after the space, so nothing else would ever clean them), and because a truncate bypasses the trigger that copies importance scores onto findings, those scores are reset to 0 explicitly, again block range by block range.
• VACUUM (ANALYZE) closes every wipe. DELETE leaves dead tuples behind: the rows are gone but the disk stays full and the planner keeps reporting the old counts. The closing pass reclaims the pages and refreshes the estimates, so the overview shows the effect immediately. (Plain VACUUM, never FULL — no blocking lock. The TRUNCATE and rewrite paths free heap and indexes instantly; on the DELETE paths, heap pages are reclaimed for reuse while heavily bloated indexes only shrink on rebuild, so a residual size may remain visible.)
• Progress you can poll. POST /maintenance/cleanup/:key answers 202 { runId }; GET /maintenance/cleanup/runs/:runId reports processed, currentTable, a note while the run waits or drops indexes, and the final result. Run records live in API memory (last 50) — an API restart drops them, in which case the overview itself shows the outcome.
• Live work is skipped, never deleted. Running or pending scans, queued or active jobs and unfinished transfers are counted under “left alone” in the result instead of being touched.
• Foreign keys are honoured in dependency order. Children before parents (logs before runs, members before clusters, chunks before jobs); nullable references are nulled first (assets pointing at a deleted run); database cascades and SET NULL rules do the rest. Cleanup cannot strand dangling references.
• Only allow-listed tables are ever named. Identifiers are validated before they reach SQL, and protected datasets have no cleanup key — an unknown key is a 404, not a wipe.
• Allowed while paused — that is the point. Cleanup is the one mutation a frozen workspace still accepts: pause stops every writer, so the wipe drains without anything re-inserting behind it. The recommended flow is pause → clean → resume. (Demo mode still blocks it entirely.)

Row counts are planner estimates and may read -1/stale right after massive writes; sizes include indexes and TOAST. Both are measured live on each visit to the tab — there is no cache to go stale.