MODULE 05 OF 12 · RELEASE 2

Production RAG & Knowledge Systems

Letting your AI look things up in your own documents — without handing someone the wrong file.

So what is this really about?

You can let an AI answer from your own documents by looking up relevant passages first and letting it read only those. The hard part is not the looking up — it is making sure the lookup respects who is asking, and that you can actually measure whether it finds the right things.

The words you will hear

Retrieval

Finding the handful of passages worth reading, out of thousands.

Chunk

A document cut into passage-sized pieces, because whole documents are too big to read at once.

Embedding

Turning a passage into numbers so "similar meaning" becomes "close together".

RAG

The whole pattern: look it up, then answer using what you found.

By the end you will be able to

• Build a document pipeline you can re-run safely.
• Apply permissions during the lookup, not afterwards.
• Measure whether your lookup is actually finding the right things.

Going deeper

The part everyone gets working, and the part everyone skips

Letting a model answer from your own documents is one of the most satisfying things to build. You chop documents into pieces, turn each piece into numbers, find the pieces closest to the question, and hand those to the model. A weekend gets you something that genuinely works.

Then it meets a real organisation, and two questions arrive that the weekend version cannot answer. Is it allowed to show this person that document? And is it actually finding the right passages, or does it just feel like it is?

Neither question is about the clever part. Both are about the plumbing, which is why they get skipped.

Ingestion has to be boring and repeatable

You will re-run ingestion far more often than you expect: you change how you split documents, you switch embedding models, a source system exports everything again. If re-running creates duplicates, every one of those is a small disaster to clean up.

So make it idempotent from the start. Hash the content of each chunk and make that hash unique. Re-ingesting an unchanged document then changes nothing, and re-ingesting a changed one replaces only the pieces that actually differ.

Store more than the text and the numbers. Every chunk should carry where it came from, when it was ingested, and — critically — who is allowed to see it:

{
  content_hash,     // makes re-ingestion safe
  text,
  embedding,
  source_id,        // which document, for citations
  visible_to,       // the permission this chunk inherits
  ingested_at,
}

source_id earns its place the first time someone asks "where did that come from?" A retrieval system that cannot cite is very hard to trust and impossible to debug.

Filter first, then rank

The instinct is to find the best matches and then remove the ones the user may not see. That is the wrong order, and the reason is subtle enough to be worth spelling out.

If you filter afterwards, there is a window where restricted content exists in your process, sitting in a variable, one careless log line or error message away from leaking. Worse, if any of it reaches the model before you filter, it can influence the answer even when the passage itself is never shown. The model saw it. The wording reflects it. Nobody can point at the leak.

Filtering inside the query closes the whole category. Restricted chunks are never candidates, never scored, never in memory, never in the prompt.

This also means your permission model has to be expressible as a database predicate. If it can only be evaluated by calling three services, retrieval cannot filter first — which is a good reason to keep permissions simple enough to denormalise onto the chunk.

Two kinds of search, because each fails differently

Similarity search is good at meaning and bad at exactness. Ask about "the Q3 revenue figure" and it will cheerfully return passages about quarterly financial performance in general, including the wrong quarter.

Keyword search is the opposite. It nails an exact product code or error number, and completely misses a passage that says the same thing in different words.

Running both and combining the rankings covers more than either alone. You do not need a sophisticated fusion algorithm to start — take the top results from each, interleave them, deduplicate. Measure whether it helps. It usually does, and if it does not, you have learned something about your corpus.

The evaluation set is the difference between engineering and hoping

This is where most retrieval systems stop being improvable. Someone tries a change, spot-checks three questions, decides it feels better, and ships. The next person does the same thing in the opposite direction.

What you need is small and unglamorous: thirty to fifty real questions, each with the passages that genuinely answer them marked by a person. That is a couple of hours of work and it converts every future change from an argument into a measurement.

Two things make the set worth having. Draw questions from what people actually ask, not what you imagine they ask — real questions are shorter, vaguer, and full of internal shorthand. And keep the ones that currently fail, because a set everything passes tells you nothing.

Then run it in CI with a floor. Not to chase a perfect score, but so that a change which quietly halves relevance cannot reach production unnoticed.

What good looks like

Re-running ingestion is safe and boring. A user can never retrieve something they cannot open, and you can prove it with a test rather than assert it in a meeting. And when someone proposes a change to chunking or embeddings, the conversation is about a number rather than a feeling.

How it is put together

Ingestion normalises a document, splits it with a stated strategy, embeds each chunk and stores it with its source and permission metadata. Retrieval filters by permission first, then ranks — combining vector similarity with full-text search, because each fails on cases the other handles. Generation receives only chunks the current actor may read.

Where you start

A working vector search over one unfiltered collection, with no permission model and no evaluation.

The lab

1. Make ingestion idempotent on a content hash so re-running changes nothing.
2. Attach source and permission metadata to every chunk.
3. Apply the permission filter in the query, before ranking.
4. Add full-text search alongside vector similarity and combine the rankings.
5. Build a small labelled evaluation set and measure retrieval against it.

What goes wrong, on purpose

The scenario. A document the current user may not read is the best semantic match for their question.

What should happen. It is excluded at query time, never reaches the model, and cannot appear in the answer or a citation.

How you prove it works

• A cross-user retrieval test asserting no restricted chunk is ever returned.
• A regression test on the labelled set, failing if relevance drops below a baseline.
• A test asserting re-ingesting the same document creates no duplicate chunks.

What you walk away with

A retrieval pipeline with a permission filter proven by test, a relevance baseline in CI, and idempotent ingestion.