RAG Beyond the Demo: What Retrieval Is Really Doing

The first explanation of RAG that made sense to me was simple: give an application some documents, store them in a vector database, then search them in natural language.

It is a useful picture. It is also where most explanations stop.

When I first followed what actually happened to a document inside the system, the questions changed. What does the datastore contain? Does every word become a vector? Where does the original PDF go? What happens when a policy is replaced? Can I remove a document and every vector derived from it? If the answer is not in the documents, why does search still return something?

Those questions taught me more than the diagram did. They moved RAG from an AI concept into the sort of software-engineering problem I could inspect, test and reason about.

I built a small application to make those lessons visible for other people. You can clone the Local RAG Studio repository on GitHub and follow the same path. It runs a Next.js interface and FastAPI backend locally, stores raw files and vectors in MongoDB Atlas Local, and generates embeddings with a Hugging Face model. LangChain handles the document representation and text splitting. OrbStack provides the container runtime without requiring Docker Desktop.

The application is deliberately inspectable. You can drag in a document, watch it move through extraction, chunking, embedding and indexing, then compare a vector-only baseline with metadata filtering, hybrid retrieval, rank fusion, reranking and abstention. You can inspect the stored chunks and ranking signals, download the raw file and permanently delete the whole record.

The lesson I carried forward was that RAG is data lifecycle and retrieval engineering, not a chatbot feature.

I learned to inspect retrieval before adding generation

The name RAG includes retrieval and generation. The original RAG paper described a generator working with retrieved external knowledge rather than relying only on knowledge held in model parameters.

The first version of this project does not generate an answer. It retrieves passages, filenames, locations and similarity scores. Strictly speaking, it is the retrieval foundation on which a complete RAG answer layer can be built.

I stopped there deliberately because one of the earliest useful lessons was to inspect retrieval on its own. Add an LLM too soon and a fluent answer can hide a poor search result. Show the passages directly and you can ask the more important first question: did the system retrieve evidence that can support an answer?

A generator cannot rescue a fact that never reached its context. It can only answer from something else, decline, or invent.

This is a practical continuation of the distinction I made in what Building with AI means for software engineers. The model is one component. The work is in the behaviour of the system around it.

A document does not become one vector

The first misconception I had to clear up was the terminology.

The application stores the original uploaded bytes in MongoDB GridFS. A parser extracts the readable text. That extracted text is divided into chunks. Each chunk produces one embedding vector, and each vector produced by the default model contains 384 numbers.

Document
  -> extracted text
  -> chunk 1 -> one 384-dimensional vector
  -> chunk 2 -> one 384-dimensional vector
  -> chunk 3 -> one 384-dimensional vector

A word does not become a separately stored vector. The embedding model processes tokens internally and pools their representations into one vector for the passage. The all-MiniLM-L6-v2 model card describes the result as a 384-dimensional dense vector intended for sentence and short-paragraph tasks such as semantic search.

A query goes through the same model and becomes another 384-dimensional vector. MongoDB compares that query vector with the chunk vectors and returns the nearest ones. MongoDB's documentation is explicit about an important constraint: the query vector must match the dimensions configured in the index and must be created with the same embedding model used for the stored data.

This is why changing an environment value from 384 to 768 is not an upgrade by itself. The model determines the output dimensions. Changing models means rebuilding the vector index and re-embedding the corpus. Even two models with the same dimensions should not share an index because they did not learn the same vector space.

Storing an embedding does not make it efficiently searchable. The vector is still an array of numbers attached to a chunk. MongoDB's vector index organises those arrays so the application can compare a query embedding with the corpus and retrieve nearby chunks without scanning every stored vector individually.

This index has a different job from the application's conventional database indexes, which help list documents and locate their chunks for inspection or deletion.

One chunk becomes one vector. The embedding represents meaning; the vector index makes that meaning retrievable at scale.

Chunking is where the real design decisions begin

The project currently uses LangChain's recursive text splitter with a target of 1,000 characters and 180 characters of overlap. The splitter prefers boundaries such as paragraphs and lines before falling back to smaller units.

The overlap repeats a small amount of text between neighbouring chunks. That reduces the chance that a useful sentence loses its context because it crossed a boundary. It also means the combined character count of every chunk can exceed the original extracted-text count.

Small chunks tend to retrieve precisely but can leave out a qualification. Large chunks preserve more context but dilute the subject and use more of a future LLM's context window. A leave-policy sentence saying “25 working days” is not enough if the words “full-time employee” and “excluding public holidays” ended up elsewhere.

File structure complicates this further. A Markdown heading needs to stay with its section. A spreadsheet row needs its column headings. A PowerPoint slide needs its title. A scanned PDF may contain no machine-readable text at all, and without OCR the system cannot extract or embed what a person can see on the page.

I added a long employee-terms CSV to the repository to make this failure reproducible. Under generic character splitting, the later Portugal row is separated from the header that says which number represents probation and which represents notice. The right numbers can be retrieved while their meaning has been weakened.

The CSV parser now creates one semantic record per row and repeats each column name beside its value. That same change moved the Valencia travel-rate row above the similarly worded office guide in the enhanced retrieval tests. Ranking improved because the evidence became clearer before either retrieval method saw it.

A chunk is not merely a piece of text. It is the unit of evidence the rest of the system will be asked to trust.

Vector similarity is a useful default, not a final ranking

Semantic retrieval is the part that first makes the system feel different from ordinary search. I can ask how often a hybrid employee may work away from the office, even though the policy uses different wording, and the relevant passage can still rank highly. MongoDB describes vector search as comparing proximity in a multidimensional space so that meaning, rather than only exact text, can drive retrieval.

The application preserves this as a vector-only baseline. The embedding model creates the geometry and MongoDB ranks chunks by cosine similarity within it. The interface now shows that value as a score rather than a percentage because 0.82 is not an 82 per cent probability that the passage is correct.

The baseline made the limitation concrete. For the 2026 annual-leave question, the amendment ranked first at 0.829, the superseded 20-day policy ranked second at 0.820, and the controlling 25-day policy ranked third at 0.819. The values were close because all three passages were semantically close. Nothing in cosine similarity established which source controlled.

If I ask whether the fictional company provides dental insurance, the corpus contains no answer. Vector search still returns a remote-work passage at 0.744 because one indexed chunk must be nearest. A similarity score ranks proximity, not authority, support or truth.

I added more signals instead of trusting one score

The enhanced path starts by extracting fields such as record ID, source status and effective date during ingestion. Current-policy searches exclude superseded sources before ranking, while the interface can include them deliberately for historical questions.

A reranker is a second-stage relevance judge. Fast retrieval first produces a shortlist; the reranker then spends more computation reading each query and candidate passage together. Unlike embedding search, which compares independently created vectors, it can judge the relationship between the exact question and passage. This is the retrieve-and-rerank pattern documented by Sentence Transformers.

1. Retrieve up to 30 semantic candidates with vector search.

2. Retrieve up to 30 exact-language candidates with full-text search.

3. Combine their positions with reciprocal rank fusion, rather than averaging unlike scores.

4. Score every query-passage pair with the local cross-encoder/ms-marco-MiniLM-L-6-v2 model.

5. Convert each raw model output with a sigmoid function into a bounded score.

6. Sort the passages by that reranker score.

7. Reject the set when its best score is below the configured 0.15 threshold.

The sigmoid value is still a retrieval signal, not the probability that a passage is correct or true. The 0.15 threshold was calibrated against this fictional corpus rather than borrowed as a universal confidence value. For the unsupported dental-insurance question, it makes the interface say that the available documents do not contain sufficiently relevant evidence.

The interface exposes every stage: vector score, text score, fused rank, reranker score, source status and the final threshold decision. Better retrieval came from combining meaning, literal language, document eligibility and a second judgement.

One limitation remained visible. For the vague “Valencia allowance” question, the local pipeline retrieved the hotel, meal and bicycle interpretations but could not decide whether they belonged together or required a choice. Returning several plausible passages was not the same as resolving ambiguity.

I added an optional OpenRouter gate after reranking for that decision. When two accepted candidates have scores within 0.05, the backend sends the query and at most six evidence passages to an OpenRouter model. It requests a structured answer or clarify decision using OpenRouter's JSON-schema response format. A clarification must contain at least two options tied to retrieved evidence IDs, and choosing one in the interface reruns retrieval with the refined question.

The API key stays in the backend .env file, but the ambiguity check is no longer fully local: when it runs, the query and selected passage excerpts leave the machine through OpenRouter. Without a key, or if the external request fails, local retrieval continues and exposes the ambiguity stage as not_configured or error. The model may decide that clarification is needed, but the application still validates the shape and evidence behind that decision.

Deletion belongs in retrieval correctness

Adding a file is the obvious interface. The earlier lesson for me was that deleting one has to be treated as part of retrieval quality, not housekeeping.

An uploaded document exists in several forms: the raw GridFS file, the document metadata, extracted chunks, embedding arrays and vector-index entries. Removing only the visible document record would leave searchable information behind. Removing only the raw file would make the derived evidence impossible to audit or reprocess.

The application therefore warns the user and removes every representation. I included a manual evaluation that uploads both old and current leave policies, records the conflicting results, deletes the obsolete document through the interface, then verifies that neither MongoDB nor retrieval returns its chunks.

If a source can enter a RAG system, its update, replacement and complete removal need equally deliberate paths.

Evaluation separates a change from an improvement

One of the tempting moves in RAG is to reach for a larger embedding model when retrieval disappoints. The project uses a 384-dimensional MiniLM model and includes a commented 768-dimensional MPNet alternative.

A higher dimension can provide more representational capacity when it comes from a stronger model, but more numbers do not repair an obsolete policy, lost table heading or missing answer. It also increases storage and computation. Without a fixed test set, changing the model produces a different result without telling you whether the system became more useful.

The repository therefore contains three evaluation layers:

1. A baseline suite against current policies, procedures and reference data.

2. A conflict-and-noise suite that adds obsolete documents, amendments and ambiguous material.

3. Paired solution experiments that record a control, apply one proposed treatment and define success criteria.

The solution experiments cover metadata filtering, hybrid search, structure-aware CSV parsing, reranking, document management and minimum-score abstention. I kept the vector-only control beside the enhanced path and ran the same behavioural checks against both.

With all sample groups loaded and no OpenRouter key configured, the local pipeline passed 18 of 19 checks. It found the expected sources for direct facts and exact identifiers, excluded the superseded policy from current searches, ranked the Valencia and London rate rows above office noise, and abstained for the two missing-benefit questions. The remaining failure was the ambiguous Valencia question, reported explicitly as not_configured. I have not counted the optional model path as passing until it is exercised with a real key.

This is consistent with the practical advice in Hugging Face's Advanced RAG cookbook: create a small evaluation dataset, measure performance, and change retrieval components iteratively.

The order matters. Record the failure first. Change one component. Repeat the same question against the same corpus. Then rerun the baseline to make sure the fix did not quietly break something else.

A different result is not evidence of an improvement. A repeatable evaluation is.

Clone the project and break it deliberately

I built the repository as a learning laboratory rather than a polished claim that RAG has been solved. To run it locally:

git clone https://github.com/CodehanceHQ/rag-project.git
cd rag-project
make setup
make db-up

Then run the API and web interface in separate terminals:

make api
make web

Open http://localhost:3000. Start with the current documents under sample-documents/policies, procedures and reference. Add conflicting-versions and noise, then run the same question in vector-only and enhanced modes. The first enhanced search downloads the local reranker model.

To enable ambiguity detection, add an OpenRouter key to the root .env file and restart the API. The key remains server-side:

OPENROUTER_API_KEY=your-key-here
OPENROUTER_MODEL=openai/gpt-4.1-mini

Select Run evaluation in the retrieval screen to execute all 19 behavioural checks against the currently indexed corpus, or run the same suite from the command line. With a key configured, close-result cases can make paid OpenRouter requests.

PYTHONPATH=backend .venv/bin/python evaluations/run_retrieval_evaluation.py

The results show source ranking, metadata eligibility, abstention and the top evidence for each case. Unsupported clarification behaviour remains a failure rather than being counted as success because the system returned something plausible.

Do not upload the evaluation JSON files. They contain the expected answers, which would leak the ground truth into the material being searched.

You can also inspect MongoDB directly with Compass or mongosh. Look at the raw file metadata, chunk text and embedding arrays. Delete a document through the application and check that all of them disappear. Change the query wording. Compare a precise question with an ambiguous one. Ask something the corpus cannot answer.

The point is not to make every test pass immediately. The point is to make each failure legible enough that you know what to build next.

The test I now use for a RAG system

The first question was whether documents could be placed in a vector database and searched naturally. That question got the pipeline moving, but it was never enough to judge the result.

The questions that proved more useful were these: Which text was extracted? How was it divided? Which model created the vectors? Which source version was eligible? Why did this passage rank? What was excluded? What happens when no passage supports an answer? Can I remove every derived copy of a document? Will the same evaluation still pass after I change the system?

Those are lessons from looking beneath the first working result. They are also the difference between demonstrating vector search and building with AI.

If I cannot inspect the evidence, reproduce the failure and verify the lifecycle of the source, I do not yet have a RAG system I can trust.

Sources

Linked so you can inspect the implementation, evidence and limits.

• CodehanceHQ: Local RAG Studio repository. The application, architecture, local setup, fictional policy corpus and evaluation suites discussed in the article. It is a learning implementation and currently exposes retrieved passages rather than generating final answers.

• Lewis and colleagues: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The 2020 paper supplies the original framing of generation combined with retrieved external knowledge. Its experiments concern the authors' trained RAG models and should not be read as a performance claim for this project.

• LangChain: recursive text splitter documentation. The implementation reference for recursively dividing text into overlapping chunks. The project's 1,000-character size and 180-character overlap are local choices, not universal recommendations from LangChain.

• Sentence Transformers: all-MiniLM-L6-v2 model card. Confirms that the default local model maps sentences and short paragraphs into a 384-dimensional dense vector space and documents its intended semantic-search use and input limitations.

• MongoDB: Vector Search overview. Documents semantic vector retrieval, metadata pre-filtering and the relationship between query vectors, stored embeddings and the vector index. The application uses these capabilities locally, but its evaluation results apply only to the included fictional corpus.

• MongoDB: combine semantic and full-text results with reciprocal rank fusion. Supplies the rank-fusion mechanism used to combine candidate lists without treating vector and text scores as directly comparable.

• Sentence Transformers: retrieve and rerank. Documents the two-stage pattern of fast candidate retrieval followed by a slower CrossEncoder. The project's chosen model and 0.15 threshold are local implementation choices, not recommendations from this guide.

• OpenRouter: API quickstart. Documents the server-side chat-completions endpoint and bearer-key authentication used by the optional ambiguity gate. Calls leave the local application and are subject to the selected provider's availability, handling and charges.

• OpenRouter: structured outputs. Documents the JSON-schema response format used to constrain the ambiguity decision. A valid schema does not establish that the model's decision is correct, so the application also checks option count and retrieved evidence IDs.

• MongoDB: vectorSearch operator reference. Confirms that query dimensions must match the index and that stored data and queries must use the same embedding model; it also documents candidate tuning and pre-filter behaviour.

• Hugging Face: Advanced RAG with LangChain. Provides the practical recommendation to establish an evaluation dataset and improve retrieval iteratively through measured changes. It is a tutorial built around a different corpus and vector-store implementation, so its examples are guidance rather than benchmarks for this project.