Embeddings
Open Harness AI → Embedding to see and change how this workspace turns evidence into meaning. Whether embeddings run at all is a switch in Settings → Cleanup › Features, next to duplicate detection; this page shows its state and links there.
Embeddings live beside the Harness rather than in general settings because they are the retrieval half of the same AI stack: the model chosen here is what the agents’ evidence lookups, the Assistant’s answers and every semantic search actually run against.
Detectors find matches. Embeddings are what let Classifyre reason about whether two matches are the same thing said twice, whether one is unusual for your corpus, and whether a search for “invoices from a shell company” should return a document that never uses either word.
What an embedding actually is
An embedding is a list of numbers — a vector — that stands for what a piece of content means. Content that means similar things gets similar vectors, so “meaning” becomes something the database can measure: two pieces of evidence are related when their vectors sit close together.
Classifyre embeds:
| What is embedded | Where it comes from |
|---|---|
| Evidence text of a finding | The match itself plus the text before and after it |
| Document chunks | Passages of ingested documents, split during scanning. This includes the text OCR reads out of pictures and the transcript of recordings and video |
| Pictures and recordings | Image and audio files themselves, when the model can embed them. See Pictures and sound |
Nothing is embedded that a detector did not already surface or that a scan did not already read. Turning embeddings off does not change what is scanned — it changes what can be concluded from it.
Where in a document a search matched
A document is ranked by its best-matching chunk, and a search by meaning returns
that chunk with the document: match.page (the page it is on, when the document
has pages) and match.text (the start of the passage). On the assets page it
reads as Best match · page 175 under the document’s name. For a 500-page annual
report that is the difference between “it is in there somewhere” and the page
that answers the question. Pages come from the file itself (a PDF) or from a
custom connector that separates them with a form feed (see
One document, many pages).
POST /<workspace>/search/assets
{"semantic": {"query": "Diageo operating cash flow 2026", "mode": "vector"}, "page": {"limit": 5}}
→ {"items": [{"asset": {...}, "findings": [...],
"match": {"score": 0.71, "page": 175, "text": "Consolidated cash flow statement …"}}]}What stops working without it
Three features are built directly on this layer, and all three go quiet when embeddings are off or still building:
- Importance ranking — the 0–1 score that sorts findings by “worth my time” rather than by severity label. See Ranking & the Semantic Layer.
- Near-duplicate clusters — the grouping that collapses one boilerplate template repeated 4,000 times into a single item.
- Semantic search and AI answers — searching by meaning, and the evidence the Assistant and the Harness retrieve to answer questions.
Detectors, scans, findings, assets and cases all keep working normally. What you lose is the layer that tells you which findings to open first.
Choosing a model
The Model section offers two ways to produce vectors.
Local inference
Runs on the same machine as Classifyre, using Transformers.js. No evidence text leaves the deployment, which is usually the deciding factor for regulated corpora. It costs CPU: embedding a large corpus is the heaviest sustained work the system does.
The list offers seventeen models. They fall into four groups, and the group matters more than the name:
| Group | Models | Choose it when |
|---|---|---|
| English, small | all-MiniLM-L6-v2 (the bundled default), all-MiniLM-L12-v2, bge-small-en-v1.5, bge-base-en-v1.5, gte-small, granite-embedding-small-english-r2 | The corpus and the questions are in English and indexing speed matters most. 23 to 106 MB. |
| Many languages, small | multilingual-e5-small, granite-embedding-97m-multilingual-r2 | The corpus is not all English and the machine is small. About 100 to 130 MB, and about a third of the default model’s indexing speed. |
| Many languages, large | EmbeddingGemma 300m, Harrier 270m, Voyage 4 nano, bge-m3, Qwen3 Embedding 0.6B | Search quality on real documents matters more than indexing time. 300 to 620 MB, and in our measurements 14 to 120 times slower to index than MiniLM. |
| Pictures or sound | EmbeddingGemma 2 (documents, pictures and sound), SigLIP 2 base and CLIP ViT-B/32 (pictures), CLAP (sound) | See Pictures and sound. Only EmbeddingGemma 2 is also a document model. |
An English-only model does not fail on German text. It answers, and finds little: in our own measurements the English-only models found 4 to 9 of 30 German sentences from an English question, and every multilingual model 23 or more.
EmbeddingGemma 2 puts text, pictures and sound in the same 768 numbers, in
over a hundred languages, so a question typed in words can land next to a
photograph. It is also by far the heaviest: about 314 MB to download for
text, 195 MB more for pictures and 340 MB more for sound, about one passage a
second on two CPU cores, and close to a minute for each picture. It is listed
under its ONNX conversion, onnx-community/embeddinggemma-2-ONNX, because
that is what the built-in worker can load. Typing google/embeddinggemma-2
as a custom model selects the same thing: Google’s own repository holds the
PyTorch weights only. The other models’ own repository names are accepted
the same way.
Dimensions, pooling and task prompts of a model on the list are filled in for you.
Downloaded models are kept in a cache on the pod, and the cache has a budget
(EMBEDDING_CACHE_MAX_MB, 1,500 by default, under the 2 Gi volume the Helm
chart gives it). When another model would not fit, the one used longest ago
is deleted and fetched again if a workspace returns to it.
The all-in-one Docker image ships one pinned model and runs offline. It cannot download a different local model — the settings page says so when that applies. Use a remote provider there, or run a deployment that permits model downloads.
Remote provider
Sends evidence text to an OpenAI-compatible embeddings API, using a provider you have already saved under AI Providers. The endpoint and the API key come from that provider, so the credential stays encrypted in one place and rotating it fixes every consumer at once.
Only providers marked Embedding model appear in the picker. Embeddings and chat completions are different endpoints with different model names — a chat provider selected here would fail on its first batch — so the flag is set deliberately when you create or edit the provider. If nothing is listed, the page offers a direct link to create one.
An embedding provider also stores the dimensions its model returns (and optionally the pooling strategy). Those describe the model, not the workspace, so selecting a remote provider here inherits them — there is nothing further to fill in. Use Test connection on the provider to confirm the dimensions are right: the test sends a real embeddings request and compares the width of the vector that comes back. A mismatch caught there costs a second; the same mismatch caught later fails the first batch of a rebuild, after the old corpus is already gone.
The provider test knows the difference. A provider marked as an embedding model is tested by embedding a probe string, not by asking for a chat completion — which is what an embeddings endpoint refuses, and why a working embedding credential used to test as broken.
This moves the CPU cost off your machine and generally improves quality — at the price of sending evidence text to a third party. That is a data-governance decision, not a performance one.
A server you run yourself
“Remote” means “not the built-in worker”, not “somebody else’s computer”. Any
server that answers POST /v1/embeddings in the OpenAI shape works the same
way: llama.cpp’s llama-server --embedding, Ollama, vLLM, LM Studio, Text
Embeddings Inference. Save it as an OpenAI-compatible provider with its base
URL (http://llama-embedding.my-namespace.svc:8080/v1), mark it Embedding
model, and leave the API key empty: such servers take none, and nothing
leaves your network.
That is how a larger open model is run on hardware you size separately from Classifyre. EmbeddingGemma 2 runs in the built-in worker too, but behind llama.cpp it gets its own CPU and memory, or a GPU. On four CPU cores it takes about a tenth of a second for a search and well under one passage a second while indexing, against about six a second for the bundled MiniLM. Searching is fast on a CPU; building the index for a large corpus is not.
Such a server can take pictures and sound as well: see Pictures and sound.
Task prompts
Some models are trained to be told which side of a search a text is on. They expect a short prompt in front of every stored passage and a different one in front of every question, and they retrieve worse without them: short strings that merely look like the question outrank the passage that answers it.
| Model family | Document prompt | Query prompt |
|---|---|---|
| EmbeddingGemma, EmbeddingGemma 2 | title: none | text: | task: search result | query: |
E5 (multilingual-e5-*) | passage: | query: |
| Nomic Embed | search_document: | search_query: |
A library such as sentence-transformers applies these by name. A bare embeddings endpoint embeds whatever string arrives, so Classifyre puts the prompt there itself: set Document prompt and Query prompt on the provider, exactly as the model card writes them, trailing space included. Leave both empty for a model that has none (OpenAI’s, the bundled MiniLM).
A built-in model from the list needs none of this: the prompts of
EmbeddingGemma, E5, Voyage 4 nano, Qwen3 and Harrier are known and applied for
you. The last two take an instruction in front of the question only
(Instruct: Given a web search query, retrieve relevant passages that answer the query), and nothing in front of a stored passage. Prompts go in front of
text only. A picture or a clip is embedded as it is.
The document prompt is part of what a stored vector means, so changing it rebuilds the corpus like a model change does. The query prompt only shapes the next search; nothing is re-embedded.
Pooling follows the model
A text model produces one vector per token, and pooling is how those
become one vector for the passage: the average of all of them (mean), the
first token’s (cls), or the last token’s (last_token). Which one is right
is decided when the model is trained. BGE and Granite put the sentence in their
first token, Qwen3 in its last, MiniLM and E5 are averaged. Read the wrong way,
a model still returns a vector of the right width, and retrieves worse with
nothing to say why.
Picking a model from the list sets its pooling. The Pooling field under Advanced is for a custom model; take the value from its model card.
Dimensions must match the model
Dimensions is not a preference. It is a property of the model you chose, and a mismatch fails every batch rather than degrading quietly. Picking a model from the dropdown fills in the right number automatically; if you type a custom model, you must supply the number its documentation states. For a remote provider the number comes from the provider itself.
Models trained to be shortened (Matryoshka models such as EmbeddingGemma and
OpenAI’s text-embedding-3-*) can return fewer dimensions on request, and
Classifyre sends the provider’s dimensions with every request. Whether that
does anything is up to the server: OpenAI’s and Google’s APIs honour it,
llama.cpp ignores it and returns the full vector, which the connection test
then reports as a mismatch. Set the dimensions to what the server returns.
How large a model can be is a question about the index, not about storage:
| Dimensions | What happens |
|---|---|
| Up to 2,000 | Indexed normally |
| 2,001 – 4,000 | Indexed at half precision — a negligible accuracy cost, and how 2,048- and 3,072-dimension models are served |
| Above 4,000 | Stored and searched correctly, but with no index — similarity queries scan the whole space and will be slow on a large corpus |
| Above 16,000 | Rejected; pgvector cannot store the vector |
Changing a model rebuilds the corpus
This is the part worth reading before you touch anything.
Vectors are only comparable within one model. A distance measured between a vector made by one model and a vector made by another is still a number — the system will happily sort by it — but it means nothing. There is no partial migration and no gradual cutover: the old vectors are not stale, they are incompatible.
So when you change any of these, Classifyre deletes the stored vectors and embeds the corpus again from scratch:
- Model
- Revision
- Dimensions
- Pooling
- Normalization
- Provider (local ↔ remote)
The page tells you what that costs before you confirm — how many vectors and how much disk will be deleted.
Turning embeddings off or back on is not on this list any more: it no longer implies a rebuild. See Turning embeddings off.
Neither are the Pictures and Sound switches. A picture’s vector lives in the same space as the text beside it, so turning one on adds vectors and turning it off removes them, and nothing already stored changes meaning.
A rebuild re-embeds stored pictures and clips along with the text, without a rescan, when the new model can embed them. Moving to a model that cannot deletes them: they would be bytes nothing reads.
What is deleted, and what is not
| Deleted | Kept |
|---|---|
| Stored vectors for this workspace | Every finding, asset, source and scan |
| Importance and quality scores derived from them | Cases, leads, events, notes and duplicate matches |
| Near-duplicate groupings | Detector configuration and results |
Your investigation is not lost. What resets is the ranking layer: while the rebuild runs, importance scores read as zero and near-duplicate clusters are unavailable, because the evidence they were computed from no longer exists. Both come back on their own — re-embedding is followed automatically by a recalibration pass that rewrites the scores.
It runs in the background
A rebuild walks the entire corpus, so on a real workspace it takes minutes to hours. It runs asynchronously: you can close the page, and progress is on the same Corpus panel — vectors written, batches still queued, and whether a reconciliation or recalibration pass is currently running.
You can also start one at any time with Rebuild now, which is the right move if you suspect the index is incomplete or inconsistent.
Turning embeddings off
The switch lives in Settings → Cleanup › Features, and turning it off asks how:
- Keep data is a pause. Vectors, importance rankings and document text chunks stay, and scans keep saving text chunks, so turning embeddings back on catches up without rescanning anything — the vectors are still in the same coordinate system.
- Delete data removes vectors, rankings (importance scores read 0), text chunks and the vector indexes, and new scans stop saving chunks. Turning back on re-embeds every finding from scratch; document text comes back only when sources are rescanned with Full rescan.
Either way the embedding and recalibration workers are held paused (they show as held in Settings → Workers) and semantic search, similar findings and near-duplicate text groups are unavailable until embeddings are back on. Detectors, scans, findings, assets and cases keep working normally.
While embeddings are off, the model sections of this page are hidden — there is nothing for them to act on — and the Corpus panel still shows what is stored, so the cost of a kept corpus stays visible. (The API still accepts a model change while embeddings are off; one that redefines the vector space purges the kept vectors, because they would be meaningless under the new model.)
Before this switch existed, turning embeddings off always deleted the corpus. A workspace turned off that way reads as off, data kept: its vectors are already gone, and turning back on embeds everything again.
Tuning throughput
The Throughput settings change how fast vectors are produced and never what a vector means, so none of them requires a rebuild.
They do differ in when they take hold. Batch size, worker concurrency and startup reconciliation are read once, when the embedding worker registers with the job queue — so they apply after the service next starts. The rest are read per batch and apply to the next one.
| Setting | What it does | Takes effect | When to change it |
|---|---|---|---|
| Inference batch size | Rows sent to the model at once | Next service start | Lower it if the machine runs short of memory during scans; raise it on a dedicated server |
| Worker concurrency | Embedding batches processed at once | Next service start | Raise it only when embedding is the bottleneck and the machine has headroom |
| Reconcile on startup | Embed anything missing when the service starts | Next service start | Leave on unless you are deliberately pausing the work |
| Inference threads | CPU threads per batch (local only) | Next batch | Lower it to keep the machine responsive while embedding |
| Parallel remote calls | Concurrent requests (remote only) | Next batch | Match your provider’s rate limit |
If the machine is under memory pressure, a smaller batch size is the first lever to reach for — it is the single setting that most directly bounds how much memory one inference step needs.
On a CPU the built-in worker gives the model one text at a time, whatever the batch size: the batch is how many texts travel to the worker in one message. Two measurements are behind that. The default weights are 8-bit, and with those the vector of a text depended on what else was in its batch (the same sentence came back at a cosine of 0.975 to 0.997 to itself, depending on the model). And a batch is padded to its longest text, which on two CPU threads made batches of 32 two to three times slower than the same texts one by one. With a GPU device the batch goes to the model whole.
Search index settings
Search index tunes the HNSW index that makes similarity search fast on large corpora.
| Setting | Effect |
|---|---|
| Graph degree (m) | Higher builds a denser, more accurate, larger index |
| Build quality (ef_construction) | Higher builds a better index, more slowly |
| Search breadth (ef_search) | Higher returns better matches, more slowly, at query time |
Only ef_search applies to existing data, and it applies to the very next query. The other two are baked into an index when it is built, so changing them affects indexes created afterwards — in practice, after the next rebuild.
The defaults are appropriate for corpora up to a few million vectors. There is rarely a reason to change them without a measured problem.
Defaults, and where they come from
Every setting on this page has a deployment default underneath it:
- On Kubernetes, from the Helm chart’s
api.embeddingvalues. - In the all-in-one Docker image, from the baked-in defaults sized to the container.
The Pictures and Sound switches follow api.embedding.images (on) and
api.embedding.audio (off), or the EMBEDDING_IMAGES and EMBEDDING_AUDIO
environment variables. Both only say what a workspace wants: on a text-only
model neither does anything.
The page shows that default under any setting you have overridden, along with Use default to return to it. An override applies to one workspace — other workspaces on the same instance are unaffected and continue to follow the deployment default, including when an administrator later changes it.
This is deliberate. A workspace holding a multilingual corpus can run a multilingual model without forcing that choice — and its cost — on every other workspace in the instance.
Reading the Corpus panel
| Figure | Meaning |
|---|---|
| Vectors | Embeddings stored in the active model’s space |
| On disk | Space taken by those vectors and their search index |
| Text chunks | Document passages eligible for embedding |
| Pictures and clips | Pictures and sound clips stored for embedding, and the disk they take. Shown when the workspace embeds either |
| Embeddable findings | Findings that carry evidence text |
| Ranked findings | Findings that already have an importance score |
The gap between Embeddable findings and Ranked findings is the honest measure of how far along the semantic layer is. While that gap is wide, treat importance-sorted views as partial — the Harness does the same, and stands down from conclusions that need ranked evidence until coverage is sufficient.
Vector spaces lists every coordinate system this workspace holds vectors in. After a normal rebuild there is exactly one, marked active.
Pictures and sound
Most embedding models read text and nothing else. With one of those, a picture is findable only through the words OCR reads in it, and a recording through its transcript. A photograph with no writing in it has no words, so no search finds it.
A multimodal model embeds the picture itself. It is trained so that a photograph of a stained ceiling and the sentence “water coming through the bedroom ceiling” get vectors close to each other. Text, pictures and sound all go into one vector space, which is why the same search box finds all of them and nothing else on this page changes.
Which models can do it
| Model | Where it runs | Pictures | Sound |
|---|---|---|---|
| EmbeddingGemma 2, built in | The built-in worker, no data leaves the deployment | Yes | Yes |
| SigLIP 2 base, CLIP ViT-B/32, built in | The built-in worker | Yes | No |
| CLAP, built in | The built-in worker | No | Yes |
| EmbeddingGemma 2 behind llama.cpp | A server you run, as a remote provider | Yes | Yes |
| Gemini Embedding 2 | Google’s API or OpenRouter, as a remote provider | Yes | Yes |
| Every other model on the list, and any text embeddings API | — | No | No |
SigLIP, CLIP and CLAP are a different kind of model from EmbeddingGemma 2: two separate encoders, one for a caption and one for a picture (or a sound), trained against each other. Their text side was trained on captions: SigLIP 2 reads 64 tokens and CLIP 77 and both cut a longer passage there, and CLAP’s was tuned to match descriptions of sounds. They are on the list for a workspace that is mostly pictures or recordings; for documents they are the wrong choice, and the measurements show by how much.
Two models that both “do pictures” are still two separate spaces. A picture embedded by one cannot be found by a question embedded by the other, so the pictures always go through the same model as the text.
Turning it on
Under Harness AI → Embedding → What is embedded:
| Switch | What it embeds | Default |
|---|---|---|
| Text | Documents, and the text read out of pictures, recordings and video | Always on |
| Pictures | Image files, as pictures | On, when the model can |
| Sound | Audio files, as sound: the first two minutes of each | Off |
A switch is greyed out when the selected model embeds text only. Picking a model that can see turns Pictures on with it; that is what such a model is usually chosen for. Sound stays off until you turn it on: its encoder is the largest part of the model, and recordings are already searchable through their transcript. A transcript holds the words and nothing else, though: a smoke alarm going off, water dripping, a machine left running in the background are not in it, and only the sound itself carries them. Turn Sound on when what matters in your recordings is not what is said.
For a remote provider, the provider says what its model takes. Edit it under AI Providers and set two things in the Pictures and sound block:
- How media is sent. Text embedding has one request format that every server copied from OpenAI. Pictures and sound have one per server, so it has to be named: llama.cpp server, Google Gemini API, or OpenRouter. Left at Text only, the provider is never sent a picture.
- Pictures and Sound: what the model takes.
Test connection then sends the provider a small test picture (and a test clip) as well as a test sentence, and checks that all of them come back as vectors of the same width. A server started without its vision encoder, or a wrong format, fails there and not on the first scanned photograph.
With llama.cpp, the vision and audio encoders are a separate file (the
“mmproj” file of the model’s GGUF repository). llama-server --embedding
loads it when given the repository with --hf-repo, and leaves it out with
--no-mmproj. A server without it answers text and refuses pictures.
What a scan sends, and what is kept
When the workspace embeds pictures, a scan prepares each image file once more for the model: a JPEG copy with its longest edge at 768 pixels, usually 20 to 90 KB. Vision encoders work at a few hundred pixels a side, so a larger copy would only take space. For sound, a recording is decoded to mono 16 kHz audio and cut into clips of 10 seconds, at most twelve per file: its first two minutes. A full clip is about 320 KB. Ten seconds is as much as CLAP hears at once, and short enough that a sound under somebody’s voice (an alarm going off while they talk) gets a clip, and a vector, nearly to itself; in a longer clip that is mostly speech, the vector says speech.
Classifyre keeps these prepared copies in the database, next to the text chunks. That is what lets a rebuild re-embed them under a new model without reading the sources again. The Corpus panel shows how many there are and what they weigh. The original files are not copied anywhere.
Only files that are a picture or a recording are prepared. Not yet covered: pictures inside documents (a photograph pasted into a Word file, the pages of a scanned PDF) and the frames of a video. Those are still found through the text read out of them.
Pictures that were already scanned
Turning Pictures or Sound on does not need a full rescan. The next ordinary scan of each source goes back for the files it concerns: an image file whose picture has not been sent yet is read once more, and everything else stays cached. No detector runs again and no finding changes. A source on the automatic sweep starts its sweep over so that it reaches the files it had already passed.
Reading a picture again does repeat its OCR, and a recording its transcription, so the first scan after turning a switch on takes longer in proportion to how many such files a source holds.
Turning a switch off deletes the stored copies and their vectors for that kind of content at once. Turning it back on collects them again on the next scan.
What to expect from search
Text, pictures and sound come back as one list. They are not ranked by raw score, because a model does not put them at one distance: the picture a caption describes sits further from the caption than an unrelated sentence does. Measured with Gemini Embedding 2, “a dog barking” scored 0.62 against the recording of the dog and 0.56 against a sentence about an invoice. So each result is placed by how far its score lies above the middle score of its own kind for that question: a passage above the median of a sample of the passages, a picture above the median of all pictures, a clip above the median of all clips. A kind with fewer than eight items has no middle to speak of and is read against the text’s.
Measured on a small mixed drive (38 files, six of them pictures, EmbeddingGemma 2):
- “water coming through the bedroom ceiling” returned the photograph of that ceiling in the first two results; “the pipe joint that failed under a sink” returned the photograph of the joint first. Neither photograph has a word of writing in it.
- A picture of a delivery note and the text OCR read from it came out almost on top of each other (similarity 0.89), which is the model saying they are the same thing.
- Pictures that had nothing to do with a question stayed far down the list.
And for sound, with Gemini Embedding 2 and two voicemails whose words never name what matters in them:
- A neighbour says “it’s started again, just listen” and holds the phone up to water running in her hallway. Text search for “water running” did not return her message in the first twelve: its transcript has none of those words. With Sound on it came first by meaning, and third in the search box, after two sentences that contain the words.
- A caretaker asks “can you hear that?” over a smoke alarm. That one did not stand out: to the model, an alarm going off in a stairwell sounds close to a baby crying and a rooster, and both were in the same drive. Distinct sounds are found; sounds that resemble others in the corpus are ranked among them.
Pictures appear in Assets search and wherever the Assistant and the Harness
look for assets by meaning. Findings are still text: a picture gets a finding
when a detector reads or classifies it, not from its embedding, so a
photograph with no writing typically has none. The Assets page lists assets
without findings by default; an API search has to ask for them with
options.includeAssetsWithoutFindings.
What it costs
Speed is the price. Measured on one CPU-only machine (an Apple M1 Pro, the model limited to a few cores), rebuilding a workspace of 252 texts and six pictures:
| The 252 texts (finding contexts and passages) | Each picture | |
|---|---|---|
| EmbeddingGemma 2, built in, 2 CPU threads | about 5 minutes, loading the model included | about 50 seconds |
| EmbeddingGemma 2 behind llama.cpp, 4 CPU threads | under 3 minutes | about 10 seconds |
A drive with thousands of photographs is hours of indexing on a CPU, once. Searching stays fast: a search embeds one short question, never a picture.
Memory grows with what is loaded. The built-in worker held about 1.4 GB with the text model and the vision encoder loaded and about 1.7 GB with the audio encoder as well; a llama.cpp server with both encoders held about 1.9 GB. The encoders are loaded only by the process that embeds scanned content; answering searches needs the text model alone.