Vector RAG (top-k retrieval)

What it is

Vector RAG turns your documents into numbers called vectors. When a user asks a question, it turns the question into a vector, too. It then finds the most similar document chunks (the top-k nearest neighbors). These chunks are added to the model's prompt for context. The model doesn't memorize the documents. It only sees the chunks provided for that specific question. In our test, we used an all-MiniLM search_memory tool to find the top chunks from our notes.

Setup

1. Chunk — Split your documents into smaller passages. You can split them by section or into fixed-size pieces. The size of these chunks is the most important factor for quality.
2. Embed — Turn each chunk into a vector using an embedding model like all-MiniLM-L6-v2. Store this vector along with the original text and its details, like its file path and timestamp.
3. Store — Load the vectors into a special database called an index. Examples include FAISS, Chroma, pgvector, Qdrant, or Pinecone.
4. Retrieve — When a user asks a question, embed it. Find the most similar chunks (the top-k). Then, add them to the prompt. If a document changes, you must update the index. Otherwise, your information will become outdated.

Architecture sketch

repo/
├─ docs/                    # the corpus: notes, decisions, PRDs, threads
├─ ingest/
│  ├─ chunk.py              # docs → passages (fixed | semantic | by-heading)
│  └─ embed.py              # passages → vectors (e.g. all-MiniLM-L6-v2)
├─ store/index              # {vector, text, meta:{path,heading,mtime}}
├─ retriever.py             # query → embed → top-k ANN (+ BM25 hybrid + rerank)
└─ agent.py                 # search_memory(query) tool → top-k chunks → prompt → LLM

The model only sees the retrieved chunks, not all your documents. This helps it handle large amounts of information. But it's also a risk. If the right answer isn't in the top few chunks, the model will never see it.

Pros & cons

Common user feedback

This feedback comes mainly from Hacker News. The point about 'top-k' instability is from a research paper. We also cite Cline, a company that sells an alternative to Vector RAG. They are a biased source, but many others share their complaint.

• Similarity finds fragments that miss the point. An engineer at Cline said: “vector similarity often retrieves fragments that mention the right keywords but miss the actual implementation logic.” (HN, vendor-flagged).
• The best chunk can get a low similarity score. Someone on Hacker News wrote: “you leave perfectly cromulent results on the table because they didn’t have the right cosine score in a vacuum.” (HN) — this is the classic problem of finding a similar but incorrect document.
• It can return old information. When asked “what happened last week in the NFL,” one tool “told me about a Buffalo Bills game from last year.” (HN).
• 'Top-k' is an unstable setting, a research paper found. The authors wrote: “We were unable to find an optimal value of K… we must conclude that this additional hyperparameter is unstable. This is a downside of using RAG in practice.” (Ovadia et al., arXiv).
• The system is designed in a way that creates stale data. As Cline's blog says: “Every merge is a potential divergence between reality and your AI’s understanding.” (Cline blog, vendor-flagged). Keeping the index up to date is a constant effort.

How it did in our benchmark

We tested this using only a search_memory tool with an all-MiniLM model. The score shows how many of the 6 facts were current. We averaged the results of 3 runs.