The constraint that pushes most teams toward local RAG is boring and non-negotiable: the documents can't leave the building. Internal wikis, support tickets, contracts, patient records — anything under a data-residency rule or a security review makes "just call a hosted embeddings API" a non-starter. The good news is that the entire retrieval-augmented generation pipeline now runs on hardware you already own, and the open models are close enough to the commercial ones that the gap rarely decides the outcome. Retrieval quality is decided by your chunking and your reranking, not by which API you didn't call.
Here's the pipeline, stage by stage: chunk the documents, embed the chunks, store the vectors, retrieve against a query, rerank, and hand the survivors to a model. Every stage has a real decision in it.
Chunking: the stage that actually moves the needle
You don't embed whole documents. You split them into passages small enough that a vector captures one idea, large enough that the idea is intact. Get this wrong and no amount of model quality saves you.
Start simple and measure before you get clever. A recursive character splitter at 400–512 tokens with 10–20% overlap is the default that works across most corpora. The splitter tries to break on paragraph boundaries first, then sentences, then words — so it respects structure instead of slicing mid-sentence. The overlap (50–100 tokens for a 512-token chunk) is insurance: if a key sentence lands on a chunk boundary, both neighbors still carry the full thought.
Smaller chunks (128–256 tokens) sharpen recall for factual lookups; larger ones (512–1024) hold together for analytical questions where context matters. Semantic chunking — grouping sentences by embedding similarity rather than length — buys you maybe 2–3 points of recall, at real compute cost. Recent analysis found plain sentence chunking matched semantic chunking up to roughly 5,000 tokens. Reach for the fancy splitter only when your evaluation numbers say the simple one is leaving recall on the table.
Embeddings: pick one open model and never mix
An embedding model turns text into a vector. The single rule you cannot break: index and query with the same model. Two models produce two incompatible coordinate systems, and cosine similarity between them is noise.
For a local stack, nomic-embed-text is the safe default — 137M parameters, a ~274 MB download versus multi-gigabyte chat models, 768 dimensions, and it holds its own against text-embedding-3-small on retrieval. If you need multilingual coverage or top-of-leaderboard recall, bge-m3 (568M, 100+ languages) is the step up; mxbai-embed-large (335M) sits in between.
Two traps specific to running these locally:
- Task prefixes are load-bearing. Nomic was trained for asymmetric retrieval, which means documents and queries get different prefixes: embed chunks as
search_document: <text> and embed the user's question as search_query: <text>. Skip the prefixes and retrieval quietly degrades — nothing errors, the numbers just get worse.
- The context window default is a footgun.
nomic-embed-text natively supports 8192 tokens but is often served with a 2048-token window. Feed it a longer chunk and the tail is silently truncated before it's ever embedded. Raise the context to 8192 if your chunks run long.
The vector store: Qdrant, Chroma, or your existing Postgres
This is the part people overthink. Three honest choices:
- Chroma — for prototyping and small corpora. Runs in-process, near-zero setup, gets you to a working retrieval loop in an afternoon. Don't ship it as your production backbone.
- Qdrant — the production default for a single-node local deployment. Written in Rust, fast, memory-efficient, with payload filtering and a clean API. This is what I reach for when the prototype has to become a service.
- pgvector — if you already run Postgres, add the extension and stop. One less system to operate is worth a lot, and HNSW indexing in pgvector is entirely adequate for most corpora.
Milvus and Weaviate exist for billion-vector scale. If you're asking whether you need them, you don't.
Wiring it together
Here's the whole loop against a locally-served embedding model and Qdrant. I'm using Ollama to serve the model — one process, an HTTP endpoint, no API key, nothing leaves the host.
import ollama
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
client = QdrantClient(host="localhost", port=6333)
client.recreate_collection(
collection_name="docs",
vectors_config=VectorParams(size=768, distance=Distance.COSINE),
)
def embed(text: str, task: str) -> list[float]:
# task is "search_document" for chunks, "search_query" for questions
resp = ollama.embeddings(model="nomic-embed-text",
prompt=f"{task}: {text}")
return resp["embedding"]
# Index: one point per chunk, original text kept in the payload
points = [
PointStruct(id=i, vector=embed(chunk, "search_document"),
payload={"text": chunk})
for i, chunk in enumerate(chunks)
]
client.upsert(collection_name="docs", points=points)
# Retrieve: same model, query prefix, ask for extra candidates to rerank
hits = client.search(
collection_name="docs",
query_vector=embed(question, "search_query"),
limit=20,
)
Note the limit=20. You retrieve wide, then narrow — which is the next stage.
Rerank, then generate
Vector search is fast but approximate; it ranks by embedding proximity, which isn't the same as relevance. So pull 20–50 candidates and re-score them with a cross-encoder — a model that reads the query and each passage together rather than comparing two independent vectors. bge-reranker-v2-m3 runs locally and typically lifts precision 15–30% for ~100–300 ms of added latency. Keep the top 3–5.
Then the generation step is almost anticlimactic. Concatenate the surviving passages into a context block, prepend the question, and send it to a local chat model — Llama 3, Qwen, Mistral, whatever fits your VRAM, served by the same Ollama process. Instruct it to answer only from the provided context and to say when the context doesn't contain the answer. That instruction is what separates a grounded system from a confident liar.
Where local wins, and where it doesn't
Local RAG wins decisively on data residency, on cost once query volume is steady (you pay for hardware once, not per token forever), and on air-gapped or offline deployments where a hosted API simply isn't reachable. You also get predictable latency and no vendor deprecating your embedding model out from under an index you can't cheaply rebuild.
What you're signing up for in exchange: you own the ops. Embedding throughput is now your GPU's problem, re-indexing 40,000 documents takes real wall-clock time, and there's no autoscaler. For a spiky prototype or a tiny corpus, a hosted API is genuinely less work and you should use one.
The concrete takeaway: stand up the dumbest version first — Chroma, nomic-embed-text with the right prefixes, recursive 512-token chunks, no reranker — and build an evaluation set of real questions with known-good answers before you tune anything. Then add the reranker, adjust chunk size, and swap the embedding model one change at a time, watching the eval numbers. Every quality decision in local RAG is measurable on your own corpus, which means you never have to guess.
Sources: Ollama embedding models, nomic-embed-text model card, Nomic Embed technical report, Best chunking strategies for RAG (Firecrawl), Chunking strategies for RAG (Weaviate), Open-source embedding models (BentoML), Vector databases for RAG (ZenML), Reranking for RAG with cross-encoders