Embeddings and the Geometry of Similarity

The drawer is a decision, not a discovery

A herbarium is a store ordered by similarity. Two pressed specimens share a drawer because somebody’s scheme said they belonged together, and the whole utility of the collection rests on that scheme being a reasonable one. Nothing in the plants themselves opens the drawer. The classification is a human artefact, made for a purpose, and it can be wrong in ways that are invisible until someone goes looking for a specimen and finds it filed under a genus it was reassigned out of forty years ago.

An embedding is the same kind of object. It is a store ordered by similarity, where the ordering was produced by an optimisation procedure rather than a taxonomist, and where the scheme is not written down anywhere in readable form. The thesis of this article is a single sentence: similarity in embedding space is a learned artefact of a training objective, not a property of meaning. Almost every retrieval failure that surprises a practitioner — the query that returns confident nonsense, the document that is never retrieved, the domain where the whole system degrades at once — follows from having forgotten that.

The forgetting is easy, because the interface encourages it. A vector database exposes a function that takes text and returns neighbours, and the neighbours are usually good. The abstraction leaks only under load, at the edges of the training distribution, and in exactly the cases a production system is most likely to encounter. What follows works through the geometry from the bottom: what an embedding is, what the conventional metric assumes, what the space actually looks like, what the index does to it, and what any of it guarantees.

Distributed representation and the distributional hypothesis

Two ideas are stacked underneath every text embedding, and they are worth separating because they fail in different ways.

The first is distributed representation: an item is represented not by a symbol or a slot but by a pattern of activity across many units, so that no single dimension carries a nameable feature and similarity between items is a function of the whole pattern. The second is the distributional hypothesis: that distributional similarity correlates with similarity of meaning, so the former can be used to estimate the latter. Sahlgren’s treatment is careful about what this actually commits one to — the hypothesis is a claim about a correlation, rooted in structuralist linguistics, and it does not by itself settle either which distributional properties one should collect or in what sense the result is “meaning” at all ^[1]. A model built on co-occurrence statistics accumulated over words in context yields paradigmatic relations; one built on direct co-occurrence yields syntagmatic ones. These are different relations, and both get called “similarity”.

The practical lineage runs from count-based vector space models — surveyed comprehensively by Turney and Pantel, who set out how term–document, word–context and pair–pattern matrices each encode a different notion of relatedness ^[2] — through the predictive models that made the technique cheap. Word2vec learned dense vectors by predicting context, at a scale that had previously been impractical ^[3]; GloVe made the connection to global co-occurrence counts explicit by fitting log-bilinear structure to the corpus statistics directly ^[4]. Contemporary text embeddings are contextual and sentence-level rather than static, but the underlying bargain is unchanged: the geometry is a compressed summary of statistical regularities in a corpus, shaped by whatever loss was minimised.

This is the crucial consequence. The geometry is downstream of the objective. If the objective rewarded predicting a neighbouring token, the geometry encodes what predicts neighbouring tokens. If it rewarded pulling paraphrase pairs together, the geometry encodes paraphrase. Neither is “meaning”, and the difference between them is not a rounding error.

Why cosine, and what cosine assumes

The conventional similarity metric is the cosine of the angle between two vectors:

Cosine became conventional for reasons that are mostly good. It is scale-invariant, which matters when vector norms correlate with nuisance properties like token frequency or document length. It reduces to an inner product on normalised vectors, which is cheap and which most approximate indexes support natively. And it is what several influential embedding models were explicitly trained to make meaningful: Sentence-BERT fine-tuned siamese networks precisely so that sentence embeddings could be compared with cosine similarity, cutting a pairwise-comparison workload from roughly 65 hours to about 5 seconds while preserving accuracy ^[5].

But scale-invariance is an assumption, not a neutral choice. It asserts that magnitude carries no information you care about — that a vector’s length is nuisance rather than signal. Sometimes that is right. Sometimes magnitude encodes confidence, specificity or frequency, and normalising it away discards exactly the evidence that would have distinguished a precise match from a vague one.

The deeper problem is that cosine similarity has no privileged status in a learned space unless something made it privileged. Steck, Ekanadham and Kallus make this concrete: analysing regularised linear models, where closed-form solutions are available, they show that cosine similarity between learned embeddings can yield arbitrary and therefore meaningless similarities, because the embeddings are identified only up to transformations that the objective does not constrain, and different regularisation schemes select different ones. Their recommendation is not to use cosine similarity uncritically, and to consider training against the similarity function you intend to deploy ^[10]. The result generalises uncomfortably: deep models combine several implicit and explicit regularisers, so the degrees of freedom that cosine is sensitive to are typically neither controlled nor observed.

Say it plainly. Cosine similarity is a well-defined function of two vectors. Whether it is a well-defined function of meaning depends entirely on whether the training procedure made it one.

The cabinet is not evenly filled

The next surprise is that the space embeddings occupy is not shaped the way the mental picture suggests. The mental picture is a cloud of points spread more or less evenly around an origin. The reality is a narrow cone.

Gao and colleagues identified the mechanism in language generation models trained with weight tying: under likelihood maximisation, most learnt word embeddings degenerate and are distributed into a narrow cone, which sharply limits their expressiveness ^[7]. Ethayarajh measured the consequence directly in contextual models, finding that the contextualised representations of all words are not isotropic in any layer of BERT, ELMo or GPT-2 — meaning that two randomly chosen words can have high cosine similarity for reasons that have nothing to do with either of them ^[6]. Li and colleagues took the same observation into sentence retrieval, arguing that BERT’s sentence embedding space is anisotropic and non-smooth, and proposing a normalising flow to map it toward an isotropic Gaussian ^[8].

Timkey and van Schijndel sharpened the diagnosis further. They found that a small number of rogue dimensions — often just one to three — dominate cosine similarity in transformer representations, and that the dimensions driving the similarity measure are not the dimensions that matter to model behaviour. Simple postprocessing such as standardisation substantially recovers the underlying representational structure ^[9].

Three practical consequences follow, and all three are routinely missed. First, raw cosine scores are not calibrated across models or even across layers; a score of 0.82 means nothing absolute, and a threshold tuned on one model transfers to another only by luck. Second, the observed dynamic range is compressed: if the mean pairwise similarity of unrelated text is already high, the usable band separating relevant from irrelevant is narrow, and small perturbations move items across it. Third, cheap postprocessing is not cosmetic. Centering and standardising before comparison is not a trick to squeeze out a benchmark point; it removes a known confound.

Distances concentrate, and intuition does not survive it

Independently of anything learned, high-dimensional geometry is hostile to nearest-neighbour intuition.

Beyer and colleagues proved the canonical result. Under broad conditions on the data and query distributions — much broader than independence and identical distribution across dimensions — as dimensionality rises the distance to the nearest data point approaches the distance to the farthest. Formally, writing $d_m$ for the distance function in $m$ dimensions, $P_m$ for a data point and $Q_m$ for a query point:

for every $\varepsilon > 0$ ^[11]. The condition is on the relative variance of the distance distribution: when distances stop varying much relative to their own mean, the nearest neighbour stops being distinguishable from everything else. The authors call such a query unstable, and their empirical work shows the effect appearing with as few as 10 to 15 dimensions on both synthetic and real data — while explicitly warning that the result does not mean high-dimensional indexing is never meaningful, since particular workloads escape the conditions ^[11].

Aggarwal, Hinneburg and Keim then showed the rate depends on the norm. For uniform data under an $L_k$ metric, the expected gap between the farthest and nearest distances scales as

with $C$ constant, while typical distances themselves grow like $d^{1/k}$. Relative contrast therefore decays with dimension, and the constant shrinks as $k$ rises — which is why they conclude that $L_1$ is preferable to $L_2$ in high dimensions, $L_2$ to $L_3$, and that fractional norms preserve contrast better still ^[12].

A third effect compounds these. Radovanovic, Nanopoulos and Ivanovic showed that as dimensionality increases, the distribution of how often a point occurs among the $k$ nearest neighbours of other points becomes strongly skewed: some points become hubs, appearing in very many neighbour lists, and they characterise this as an inherent property of data distributions in high-dimensional space rather than an artefact of any particular dataset ^[13]. In a retrieval system, a hub is a document that surfaces for queries it has nothing to do with, and it will look like a bug in the ingestion pipeline.

The honest framing is not that high-dimensional retrieval is impossible — production systems demonstrably work. It is that the intrinsic structure of the data, not the nominal dimension of the vector, is what makes retrieval meaningful, and that structure is an empirical property of your corpus that you have to check rather than assume.

Indexes that deliberately do not look at everything

Exact nearest-neighbour search over a large corpus means scanning it. Every deployed system therefore uses an approximate index, and every approximate index makes the same trade explicit: it visits a small fraction of the store, and the parameters that make it fast are the same ones that decide how often it misses.

The two dominant families work differently. Graph-based indexes, of which Hierarchical Navigable Small World is the standard, build a layered proximity graph in which the maximum layer containing an element is drawn from an exponentially decaying distribution, then greedily descend it; the structure gives logarithmic-complexity search and is robust on highly clustered data ^[14]. Quantisation-based indexes compress instead. Product quantisation decomposes the space into a Cartesian product of low-dimensional subspaces, quantises each separately, and represents a vector by the short code of its subspace indices, so that Euclidean distance can be estimated directly from codes; combined with an inverted file the method scales to corpora of billions of vectors ^[15]. Practical large-scale systems compose the two, and the engineering effort goes into keeping the memory hierarchy busy — the GPU work behind Faiss reported an implementation 8.5 times faster than the prior state of the art ^[16].

What matters conceptually is that both families expose the trade as a tunable curve rather than a hidden default. Graph indexes trade construction degree and search breadth against recall; quantisation trades code length against distance-estimate fidelity. The benchmarking literature is built around exactly this: ANN-Benchmarks evaluates algorithms as recall-versus-queries-per-second curves under a standard interface, because a single accuracy number or a single latency number is not a description of an index at all ^[17].

This is the one place in the stack where the approximation is honest. An index that reports 0.92 recall at ten thousand queries per second is telling you, in plain terms, that eight per cent of the time it does not return what an exhaustive scan would have returned. The trouble is that this number is usually measured against the exhaustive scan of the same embedding. It says nothing about whether the exhaustive scan would have returned the right document.

An objective is not a purpose

Which brings us to the transfer problem. An embedding model is fitted to minimise some loss. Retrieval is a different task, and the mismatch has a specific shape.

Dense retrieval is trained contrastively: a query and its relevant passage are pulled together, and the query is pushed away from negatives. Dense Passage Retrieval established that a dual encoder trained this way on a modest number of question–passage pairs could outperform a strong lexical baseline on open-domain retrieval, with the choice of negatives mattering substantially ^[18]. That last clause is the whole problem. The model only learns to separate a query from the negatives it was shown, and at inference time it is asked to separate that query from an entire corpus.

Xiong and colleagues named the mismatch precisely: the negatives used in training are not representative of the irrelevant documents encountered at test time, so the training distribution and the retrieval distribution diverge. Their remedy — drawing negatives from an approximate nearest neighbour index over the corpus, refreshed asynchronously as the model learns — closes the loop by making the training signal come from the same geometry the system will actually search ^[19]. That fix is instructive precisely because it was necessary: the naive objective produced a space whose local structure near a query was never optimised, which is the only structure retrieval uses.

The same reasoning applies to every off-the-shelf embedding. A model fitted for semantic textual similarity is optimised for symmetric similarity between two short sentences. Retrieval is asymmetric — a short query against a long passage — and the two do not have the same optimum. A model fitted on web-scale paraphrase data has learned that documents about the same topic are near each other, which is not the same as learning that a document answers a question.

The unfamiliar word and the degenerate region

The failure mode that surprises people most is domain shift, and it has a geometric explanation.

An embedding model handles unfamiliar vocabulary by decomposing it into subword units it has seen and composing something from them. When the input is far outside the training distribution — an internal product code, a rare surname, a specialist clinical or legal term — the composition is not grounded in anything the model learned about that token, and the resulting vectors tend to land in a region that carries little discriminative information. Everything unfamiliar looks alike, because the model has no basis on which to make it look different. The retrieval system does not fail loudly; it returns fluent, plausible, wrong neighbours.

The evidence for this is strong and specific. BEIR assembled eighteen heterogeneous retrieval datasets to test zero-shot generalisation and found that BM25 remains a robust baseline, that re-ranking and late-interaction models achieve the best zero-shot performance but at high computational cost, and that dense retrievers are computationally efficient but often underperform out of domain, leaving considerable room for improvement in generalisation ^[20]. Sciavolino and colleagues isolated the mechanism: on a benchmark of simple entity-centric questions built from Wikidata, dense retrievers drastically underperformed sparse methods, generalising poorly to rare entities unless the question pattern had been seen in training, and data augmentation did not fix it ^[21].

Read those two results together and the operational rule is unavoidable. A dense retriever is a component with a domain of validity, and rare proper nouns sit outside it. The standard mitigation — hybrid retrieval, combining a lexical index with a dense one — is not a hedge or an engineering compromise. It is the correct response to a known and measured failure region, because exact lexical matching does not degrade on unfamiliar tokens in the way that learned composition does.

Measure the drawer, not the essay

The last error is evaluative, and it is the most expensive because it hides all the others.

In a retrieval-augmented system, retrieval and generation are separate components in series ^[24]. Judging the system only by the quality of its final answer conflates them, and the conflation is asymmetric: a strong generator can produce a satisfying answer from mediocre passages, masking a retrieval defect until the day a question arrives that the model cannot cover for. Conversely, a retrieval improvement can fail to move end-to-end scores at all, which leads teams to abandon the work that was actually correct.

Retrieval must therefore be measured on its own terms, against its own labelled relevance judgements, with rank-aware metrics — recall at $k$ for whether the evidence is present at all, and a graded measure such as normalised discounted cumulative gain for whether it is ranked usefully. BEIR’s use of nDCG at 10 across heterogeneous domains is the standard reference for how to do this out of domain ^[20]. For embedding models generally, MTEB spans eight task types over 58 datasets and 112 languages, and its headline finding is the one that matters here: across 33 benchmarked models, no single method dominated across all tasks ^[22]. There is no best embedding, only a best embedding for a task and a corpus.

Above that, the generation stage needs its own instrumentation. RAGAs proposes reference-free metrics that separately score the retrieval and generation aspects of a pipeline — faithfulness of the answer to retrieved context, and the relevance of the retrieved context to the question — precisely so that the two can be attributed independently ^[23]. Whether one adopts that specific framework matters less than adopting the discipline: three measurements, not one. Did the index return what an exhaustive scan of the same embedding would return? Did that exhaustive scan contain the right evidence? Did the generator use it?

Those three questions correspond to three distinct fixes — index parameters, embedding model or hybrid strategy, and prompt or model choice — and a single end-to-end score cannot tell you which one to reach for.

Predictions, with the observations that would falsify them

These are forecasts, separated from the sourced analysis above. Horizon: 8 August 2029.

One. Hybrid lexical-plus-dense retrieval will remain the default production architecture rather than being superseded by a purely dense system, because the out-of-distribution failure on rare entities is a property of learned composition rather than of any particular model’s capacity. Disconfirmed if a widely reproduced evaluation shows a purely dense retriever matching or beating hybrid retrieval on entity-centric out-of-domain benchmarks without lexical assistance.

Two. Similarity-function-aware training — fitting the model against the metric and the index it will actually be served with, in the manner of approximate-neighbour negative mining — will become standard practice for production retrievers rather than a specialist technique. Disconfirmed if leading open embedding models continue to be released with no statement of the intended similarity function and no index-aware training stage.

Three. Published embedding evaluations will increasingly report retrieval quality separately from end-to-end generation quality, because the two have been shown to move independently. Disconfirmed if the dominant public leaderboards in 2029 still rank retrieval-augmented systems on a single blended answer-quality score.

Four. Reported similarity thresholds will not become portable. There will be no accepted cross-model calibration that makes a cosine score of, say, 0.8 mean the same thing in two different embedding spaces. Disconfirmed if a calibration standard is adopted across at least three major embedding providers.

None of these requires a breakthrough or a reversal. They follow from structure already visible in the published record: an anisotropic space, a metric with no privileged status, an index that trades recall for speed, and an objective that is not the task.

What to take away

An embedding is a filing scheme. It was produced by an optimisation procedure with a particular objective over a particular corpus, and it inherits that objective’s blind spots exactly as a herbarium inherits its taxonomist’s. The geometry is real and useful, and it guarantees far less than the interface implies: cosine similarity is privileged only if training made it so, the space is crowded rather than evenly filled, distances concentrate as dimension rises, and the index deliberately does not look at everything.

Practically, four habits follow. Check the intrinsic structure of your own corpus rather than trusting the nominal dimension. Centre and standardise before comparing, and never treat a raw similarity score as portable. Keep a lexical index alongside the dense one, because the failure region on unfamiliar vocabulary is measured, not hypothetical. And instrument the retrieval stage separately from the generation stage, because otherwise you will be debugging the wrong component.

The specimens in a drawer are not similar. Someone decided they were. Every question worth asking about a retrieval system is a question about who decided, on what evidence, and whether the decision still holds for the thing you are looking for now.