RAG — how AI uses your knowledge | How AI Works

01 · End to end

Watch your question find its documents

in one sentence: embed the question, fetch the closest by , paste them into the prompt, answer from them. Run it — the similarity scores below are computed live from real vectors.

Your knowledge base: 12 documents · pick a question to run the pipeline

Q3 Financial SummaryFY Budget MemoRemote Work Policy v5PTO & Leave GuideProduct Roadmap H2Pricing Page CopyIncident Runbook — PaymentsAPI Style GuideCompetitor BattlecardData Processing AgreementSecurity OverviewOffice Move FAQ

02 · The unglamorous part that decides quality

Chunking: slice too thin or too thick and retrieval dies

Drag the sliders over a real document. The query's answer lives in one sentence — watch what chunk size and overlap do to its findability.

Onboarding Handbook (excerpt) · query: "What's the retirement plan match?"

Chunk size120 tok

Overlap15%

chunk 1

Welcome to the team. Your first week is about access and context, not output. On day one, IT will issue your laptop and SSO creden…

chunk 2 · contains answer

…ion are opt-in. The retirement plan matches 50% of contributions up to 6% of salary, vesting immediately. For expenses, use the corp…

chunk 3

cs: code review is required for every change, no direct pushes to main. On-call starts only after your third month, always paired …

3 chunks · Good retrieval

The answer sits in a focused chunk with enough surrounding context to embed well and to let the model interpret it. This is the zone most production systems target.

03 · Production RAG

Where naive RAG breaks — and the fixes

Pure embedding search has blind spots. The classic: exact identifiers. Open Engineer depth (toggle, top right) for the full production picture — or skim the failure gallery below.

Stale index

The policy changed Tuesday; the index was built Monday. Retrieval confidently returns the old truth. Mitigation: event-driven re-indexing, freshness metadata.

Bad chunking

The answer straddles a chunk boundary: 'matches 50%…' in one chunk, 'of salary' in the next. Neither chunk ranks. Mitigation: overlap, semantic chunking.

Lost in the middle

The right chunk was retrieved — as result #7 of 20. Models attend most to the start and end of context; the middle gets skimmed. Mitigation: rerank, then keep top 3–5.

Query–document mismatch

Users ask 'can I bring my dog to work?'; the policy says 'animals are prohibited on premises'. Different words, same meaning — embeddings usually bridge this, keywords never do. Mitigation: query rewriting, hybrid search.

04 · Choosing an architecture

RAG, fine-tuning, or long context?

Three questions usually settle it.

Does the knowledge change often?

Do answers need citations to sources?

How big is the knowledge base?

Knowledge check

Did it stick?

1/3Why must documents and queries use the same embedding model?

2/3Your RAG system can't answer 'what is error E-1234?' even though a runbook covers it. Likeliest cause?

3/3Why do too-large chunks hurt retrieval?

Loading module…

Common questions this module answers

What is retrieval-augmented generation (RAG)?

RAG retrieves relevant documents from your own data using embedding similarity, pastes them into the model's prompt as context, and instructs the model to answer only from that context — grounding answers in facts the model was never trained on.

When should I use RAG instead of fine-tuning?

Use RAG for knowledge that changes (your docs, recent data) and when you need citations. Use fine-tuning for style, format and behavior. Many production systems combine both.

What chunk size should I use for RAG?

A common starting point is 200–400 tokens with 10–20% overlap. Too-small chunks lose surrounding context; too-large chunks dilute the relevant passage and waste context window.