Here is an experiment worth sitting with. Take a corpus, a set of queries, and the exact correct documents for each query. Now throw out training and generalization entirely: optimize the embedding vectors directly against the known answers, with full access to the test set, using gradient descent until the loss stops moving. You are letting the model cheat as hard as mathematically possible, memorizing which documents belong to which query.
Past a certain corpus size, it still fails.
That result comes from "On the Theoretical Limitations of Embedding-Based Retrieval" (Weller, Boratko, Naim, and Lee, 2025; accepted at ICLR 2026), and it should change how you reason about retrieval systems. The failure has nothing to do with model quality, training data, or clever loss functions. It is geometry, and it is decided the moment you pick an embedding dimension.
The constraint is a matrix rank
Strip a single-vector retriever down to what it actually computes. Every query becomes a vector, every document becomes a vector, and relevance is a dot product. Score the whole corpus against the whole query set and you get one matrix:
import numpy as np
# A single-vector retriever scores every (query, doc) pair
# as a dot product in d dimensions.
Q = np.random.randn(n_queries, d) # query vectors
D = np.random.randn(n_docs, d) # document vectors
scores = Q @ D.T # shape: (n_queries, n_docs)
# However you train Q and D, this matrix factors through an
# inner dimension of d — so rank(scores) <= d. That one number
# caps which top-k document *sets* can ever be produced.
assert np.linalg.matrix_rank(scores) <= d
The score matrix factors through d, so its rank can never exceed d. And the set of "which documents rank above which" patterns you can express is governed by the sign-rank of the relevance matrix — the smallest dimension in which you can place points so that every required ordering comes out with the correct sign. The paper formalizes this as the row-wise order-preserving rank and proves it is pinned between rank±(2A−1) − 1 and rank±(2A−1), where A is the ideal query-document relevance matrix.
The consequence is blunt: the number of distinct top-k document subsets a d-dimensional embedding can return, across all possible queries, is capped. Not "hard to learn." Not "needs more data." Structurally unreachable, because there is no arrangement of vectors in d dimensions that produces those orderings at all.
Putting a number on it
The authors ran the free-embedding experiment above for k = 2 and measured the critical-n: the corpus size at which you can no longer realize every possible top-2 pair. The scaling is steep at small dimensions and revealing:
Embedding dimension d |
Critical corpus size |
| 4 |
10 |
| 10 |
36 |
| 20 |
120 |
| 30 |
261 |
| 45 |
626 |
They fit a cubic to these points — y = −10.5322 + 4.0309d + 0.0520d² + 0.0037d³, with r² = 0.999 — and extrapolated to the dimensions real models actually ship with. The numbers land in a range that feels uncomfortably close to production:
- d = 512 → roughly 500,000 documents
- d = 768 → roughly 1.7 million
- d = 1024 → roughly 4 million
Read those as generous upper bounds, because they come from vectors optimized directly on the test set. A model that has to generalize from training data will hit the wall far earlier. And notice the practical inversion: to represent every relevant combination across a corpus of a few million documents — routine for enterprise RAG — you already need four-figure dimensions, and the requirement keeps climbing with k and corpus size.
LIMIT: a benchmark a fifth-grader passes and GPT-class embedders flunk
To show this is not a blackboard curiosity, the paper introduces LIMIT, a deliberately trivial natural-language benchmark built to exercise the bound. The small version is 46 documents and 1,000 queries. The queries are simple — the kind of "who likes X?" relevance any human resolves instantly.
State-of-the-art single-vector models collapse on it. On the small set, recall@2 lands in the low single digits:
| Model |
Dim |
Recall@2 |
| Promptriever |
4096 |
3.0 |
| GritLM |
4096 |
2.4 |
| Gemini Embedding |
3072 |
1.6 |
| E5-Mistral |
4096 |
1.3 |
On the full 50k-document version, the best embedders stay under 20 recall@100. Meanwhile:
- BM25 — plain lexical scoring, no neural embeddings — hits 85.7 recall@2.
- GTE-ModernColBERT, a multi-vector model, reaches 23.1 recall@2: much better than single-vector, still far from solved.
- A cross-encoder (Gemini 2.5 Pro scoring query-document pairs jointly) gets 100% on all 1,000 queries.
The dimension of your embedding is not a quality knob you can turn up later. It is the ceiling on how many distinct answer-sets your retriever can ever produce, and no amount of training moves it.
The pattern is the whole point. BM25 wins because a sparse lexical vector effectively lives in a very high-dimensional space. The cross-encoder wins because it never compresses the query and document into independent vectors — it looks at them together, so no rank bottleneck exists. Every architecture that dodges the single-vector dot product dodges the limit.
What to actually do about it
This is not a reason to abandon vector search. Dense retrieval is fast, cheap, and excellent as a first-stage filter. It is a reason to stop treating one embedding as a universal relevance oracle.
- Do not size your embedding dimension by leaderboards. Size it against your corpus and how combinatorial your relevance is. If queries ask for arbitrary combinations of documents, a small vector cannot represent all of them regardless of the model.
- Put a reranker behind retrieval. Pull a generous candidate set with a bi-encoder, then rescore with a cross-encoder — it has no rank ceiling because it never factorizes the pair.
- Go hybrid. Fuse dense scores with BM25 or a learned sparse model (SPLADE-style). Lexical signal covers exactly the high-rank cases where dense vectors run out of room.
- Consider multi-vector (ColBERT-style) when late interaction is affordable — it raised LIMIT recall several-fold, at higher storage cost.
The takeaway is a design rule, not a warning: a single dense vector is a lossy projection with a countable number of expressible orderings. Retrieve with it, but never let it be the last word. Reranking and lexical fusion are not optional polish on a mature RAG stack — they are how you buy back the answers your embedding dimension mathematically cannot hold.
Sources: On the Theoretical Limitations of Embedding-Based Retrieval (arXiv:2508.21038) · Full text (HTML)