Cosine similarity is not a probability: tuning pgvector thresholds for real matching

TL;DR: While building an AI matching engine for my job board side project on Postgres and pgvector, I learned that cosine similarity scores are not probabilities, and a threshold that works for one embedding model silently returns nothing for another. With text-embedding-3-small, a perfect on-topic match lands around 0.45 to 0.55, not 0.9. I had carried over a threshold of 0.78 from an older setup, every query came back empty, and the feature looked like it was working because the LLM gracefully handled the empty case. Once I understood the problem, I laid out the realistic options: a calibrated global threshold, per-context thresholds, plain top-K with no threshold, a reranking stage, and hybrid search. I ended up with the least glamorous one, a calibrated threshold used as a noise floor under a hard top-K cap, backed by a tiny eval set of real query and match pairs. This post walks through the problem, the options, and exactly why I chose what I chose.

The bug that returned 200 OK

The feature was a retrieval-augmented chat: embed the user's question, find the most similar chunks in Postgres via pgvector, pass them to the LLM as context, answer with citations. Standard first RAG build. The retrieval function took a similarity threshold, and I had set it to 0.78 because that was the value I had seen in a popular starter template.

Every request succeeded. The endpoint returned 200, the response streamed, the tone was friendly. The answers were also useless: every single one was a variant of "the documentation does not cover this, please ask a human", which is exactly what my system prompt instructed the model to say when it received no context. The retrieval was returning zero chunks for every query, including questions with a word-for-word match in the corpus, and nothing in the stack treated that as an error. An empty result set is a valid result set. A miscalibrated threshold does not crash, it degrades, and unless you test with a query you know should hit, you will not notice.

The root cause was a broken mental model: I read 0.78 as "78% relevant", so it sounded like a reasonable bar. Cosine similarity does not work that way. It is the angle between two vectors, and where typical values land depends entirely on the geometry of the model that produced them. text-embedding-ada-002, the older generation, packs embeddings into a narrow cone: on-topic matches score 0.78 to 0.85 and even unrelated texts often exceed 0.7, so 0.78 is a sane cutoff there. text-embedding-3-small spreads its vectors far more, so every absolute score is lower. When I dropped the threshold to zero and looked at raw numbers for my corpus (roughly 70 chunks of 150 to 400 words), the structure was clear: the genuinely correct chunk for an on-topic question scored about 0.47, related-but-secondary chunks landed at 0.30 to 0.40, and unrelated noise stayed below about 0.20. My inherited threshold of 0.78 sat above the best possible score the model would ever produce. The quick way to see this yourself:

select
  left(content, 60) as preview,
  1 - (embedding <=> $1) as similarity
from chunks
order by embedding <=> $1
limit 20;

(In pgvector, <=> is cosine distance, so similarity is one minus that. The index type, HNSW versus IVFFlat, affects latency and search recall, not the similarity values themselves.)

So the concrete problem: retrieval must reliably surface the 0.47 chunk, usually keep the useful 0.30 to 0.40 band, exclude the sub-0.20 noise, never silently return nothing for an answerable question, and not stuff the prompt with garbage that costs tokens on every request.

The options on the table

Once the bug was understood, the fix was not obvious, because several standard designs solve this class of problem. These are the ones I seriously considered.

Option 1: a calibrated global threshold

Keep the single-threshold design, but derive the number from my own data instead of a template: run test queries with the threshold at zero, find the gap between relevant scores and noise, place the cutoff inside it. Pros: trivial to implement (one constant changes), zero added latency, zero added cost, easy to reason about, and it preserves the ability to return an empty set when nothing is relevant. Cons: the calibration is a snapshot, not a law. It is only valid for the current embedding model, chunking strategy, and corpus shape; change any of these and the number quietly goes stale, which is exactly the failure I had just lived through. It also applies one bar to every query, even though terse keyword queries and full natural-language questions occupy different parts of the score range.

Option 2: per-context thresholds

Different cutoffs for different situations: one per document collection, one per query type, or even one derived per query from the shape of its own result scores. Pros: strictly more precise. A heterogeneous corpus with long and short chunks genuinely produces different score bands per section, and this design can honor that. Cons: every threshold is another number that can silently go stale, and now there are N of them. Calibrating one cutoff took me a test set and an afternoon; maintaining a matrix of them needs tooling I did not have and a corpus large enough to measure each context reliably, which 70 chunks is not. Sophistication that outruns your ability to evaluate it is just a fancier way to be wrong.

Option 3: top-K with no threshold at all

Drop the threshold entirely and always take the K nearest chunks. Pros: it can never return an empty set, so the silent-refusal failure mode is structurally impossible, and it is immune to model swaps, because ranking survives even when absolute scores change scale. Cons: it can never return an empty set. For an off-topic question the corpus truly cannot answer, top-K dutifully delivers the five least-irrelevant chunks, and the LLM now has plausible-looking context for a question it should refuse. That converts my visible failure (over-refusal) into a worse invisible one (confident answers grounded in noise). It also pays for K chunks of prompt tokens on every request, relevant or not.

Option 4: retrieve wide, then rerank

Pull a generous top-K cheaply from pgvector, then let a cross-encoder reranker score each candidate against the query and keep the best few. Pros: rerankers judge actual query-to-text relevance rather than raw vector geometry, so precision genuinely improves, and reranker scores are far more comparable across queries than cosine similarity is. Cons: a second model call on the hot path, which means added latency, added per-request cost, and another vendor or self-hosted model to operate. And it does not remove the calibration question, it moves it: you still need a cutoff on the reranker's score to decide when to return nothing.

Option 5: hybrid search

Combine vector similarity with classic full-text search (Postgres has it built in) and merge the two rankings, typically with reciprocal rank fusion. Pros: lexical search catches exactly what embeddings fumble: identifiers, names, rare terms, abbreviations. For matching-style workloads full of proper nouns it is often the single biggest quality win. Cons: two query paths to maintain and tune, a fusion step with its own parameters, and again no escape from the core question of when the merged results are too weak to use.

What I chose and why

I chose option 1 with a piece of option 3 bolted on: a calibrated threshold used as a noise floor, under a hard top-K cap. Retrieval returns at most five chunks scoring above 0.3. The threshold's only job is to keep sub-noise chunks out of the slots and to preserve the ability to return nothing; the cap, not the threshold, is what bounds cost. To blunt option 1's biggest con, calibration going stale, I keep the test queries and their expected matches as a tiny eval set in the repo and rerun them whenever the model, the chunking, or the corpus changes. It is a five-line script, and it turns "retrieval feels worse lately" into a diff I can read.

The reasoning, explicitly. My corpus was small and monolingual, and the measured distribution had a wide, clean gap between relevant and noise, so the main advantage of per-context thresholds and reranking, finer discrimination, was solving a problem I could not yet detect, at a price in complexity and latency I would definitely pay. Pure top-K lost on the refusal requirement: for a chat feature that cites sources, "I do not know" is a feature, and top-K structurally cannot say it. Hybrid search was the strongest contender and the one I would reach for first as the corpus grows and identifier-heavy queries appear; I skipped it initially because my eval pairs showed pure vector retrieval already hitting the right chunk, and I had no failing case that lexical search would fix. Adding machinery without a failing case to justify it is how systems get slow before they get good.

Within the chosen design, one detail matters: place the cutoff low in the gap. The failure costs are asymmetric. A slightly-too-low threshold feeds the LLM one extra marginal chunk, which it is good at ignoring and which costs a few tokens. A slightly-too-high threshold feeds it nothing, and it either refuses or hallucinates with the confidence of a grounded answer. Err toward recall; let the cap contain the cost.

Lessons

• Similarity scores are model-specific, not probabilities. 0.47 was a perfect match for one model; 0.78 was above the other model's ceiling. Never reuse a threshold across embedding models, and never read a score as a percentage of relevance.
• Empty retrieval fails silently. 200 OK plus a polite fallback answer looks exactly like a working system. Test with queries that must hit, and watch the rate of empty-context responses.
• Enumerate the options before patching the number. Global threshold, per-context thresholds, top-K, reranking, hybrid: each fails differently. Pick based on which failure you can least afford, not on which design sounds most advanced.
• Preserve the ability to return nothing. For cited answers, refusing is a feature. Pure top-K removes it and trades visible over-refusal for invisible confident nonsense.
• Calibrate by looking, and keep the evidence. Run with the threshold at zero, place the cutoff in the gap for your own corpus, and keep the query and match pairs as a regression check for the day the model or chunking changes.
• Err toward recall, control cost with a cap. An extra marginal chunk costs tokens; a missing chunk costs correctness. Threshold as noise floor, top-K as budget.