RAG Pipeline: Architecture, Steps and Real Numbers
The 9 steps of a RAG pipeline, from loading docs to the final answer, with a framework-free build, measured costs, and the bug each step can cause.
A RAG pipeline is two pipelines. One runs once, ahead of time: it cuts your documents into chunks, turns each chunk into an embedding, and stores the vectors. The other runs on every question: it embeds the question, finds the closest chunks, pastes them into the prompt, and asks the model.
That’s the whole architecture. This post walks through every step on a real bot, DocBot, a support assistant for a made-up SaaS company with 24 help docs. Every number below comes from DocBot’s runs, not from a vendor slide.
RAG architecture in one diagram
INDEXING (runs once, and again when docs change)
docs ──► 1 load + clean ──► 2 chunk ──► 3 embed ──► 4 store
24 .md plain text 67 chunks 67 × 1,536 vectors + ids
numbers + metadata
QUERY (runs on every question)
question ──► 5 embed ──► 6 retrieve ──► 7 filter / rerank ──► 8 augment ──► 9 generate
1 × 1,536 top 4 by drop stale or one message: answer
numbers cosine off-limits chunks rules + slips + slip ids
+ question
Steps 1 to 4 decide what can be found. Steps 5 to 9 decide what the model reads. Most RAG bugs live in the first half and show up in the second.
Indexing: steps 1 to 4
1. Load and clean the documents
Read every source into plain text: markdown, PDFs, HTML pages, ticket exports. Strip navigation, cookie banners and repeated footers. They match a lot of questions and help none.
This is also where you decide what goes in. DocBot’s corpus has a 2025 refund policy (30 days) sitting next to the 2026 one (15 days). Both were loaded. That one decision causes the biggest bug in this post, at step 6.
2. Chunk: cut documents into slips
The model gets a few chunks per question, never the whole corpus. So each chunk should answer one thing on its own.
DocBot’s rule is simple: one chunk per ## section, and any section over 200 tokens is split at blank lines. Result: 24 docs → 67 chunks, median 69 tokens, smallest 15, largest 195. Each chunk keeps an id like refund_policy_2026.md#2 (file name + section number), which the model cites later.
Section-based chunking works when your docs have real headings. When they don’t (scanned PDFs, chat logs), the common fallback is fixed-size windows of a few hundred tokens with some overlap. There is no universal best chunk size. Test it on your own questions.
3. Embed every chunk
An embedding model turns each chunk into a list of numbers where similar meaning ends up close together. DocBot uses text-embedding-3-small: 1,536 numbers per chunk.
The cost is small. All 67 chunks total 5,156 tokens. At $0.02 per million tokens, embedding the whole corpus costs about $0.0001. You pay it again only for chunks that change.
4. Store the vectors
Store each vector with its chunk text, its id, and any metadata you’ll want to filter on later: document date, version, product, who may read it.
Do you need a vector database? Not at this size. Searching DocBot’s 67 vectors with plain numpy takes 4.7 µs per question. We also timed brute force on 1 million generated vectors: 28 ms per query with numpy, against 0.3 ms with an HNSW index (which still found 9,980 of the true top 10,000 neighbours). An index starts to matter somewhere past a hundred thousand chunks, or when you need filters and updates handled for you.
Query: steps 5 to 9
5. Embed the question
Same embedding model as step 3, always. Vectors from two different models live on different maps, so their distances mean nothing.
Some models also want a prefix: the e5 family expects query: on questions and passage: on chunks. Read the model card.
6. Retrieve the top k chunks
Compare the question vector with every chunk vector (cosine similarity) and keep the best k. DocBot uses k = 4.
Here is what came back for “How many days do I have to ask for a refund?”:
| Rank | Chunk | Cosine |
|---|---|---|
| 1 | refund_policy_2025.md#2 (old policy: 30 days) |
0.5650 |
| 2 | refund_policy_2026.md#2 (current: 15 days) |
0.5570 |
| 3 | refund_policy_2026.md#5 (how to request) |
0.4952 |
| 4 | refund_policy_2025.md#5 (how to request) |
0.4890 |
The stale policy ranked first. Retrieval did its job: both chunks really are about the refund window. The bug was loading the old document in step 1.
7. Filter and rerank (optional, often needed)
Two cheap fixes sit between retrieval and the prompt:
- Metadata filters. Keep only
version = current, or only docs this user may read. That one filter would have removed the 2025 chunk above. Filter inside the search when your store supports it: filtering after taking the top 4 can leave you with fewer than 4 chunks. - Reranking. Fetch more candidates (say 20), then let a cross-encoder reranker score each one against the question and keep the best 4. It’s slower than a vector lookup, but it reads question and chunk together, so it catches near-misses that cosine ranks wrong.
8. Augment: build one message
The model has exactly one input. DocBot’s message has three parts: instructions, the 4 chunks with their ids, and the question.
SYSTEM:
You are DocBot, the support assistant for Tarnik. Answer the customer briefly,
using only the slips below. Cite the slip id you used in square brackets,
like [file.md#2]. If the slips don't contain the answer, say you don't know.
USER:
Slips:
[refund_policy_2025.md#2]
## 2. Refund window
You can request a full refund within 30 days of your purchase or renewal date. ...
[refund_policy_2026.md#2]
## 2. Refund window
You can request a full refund within 15 days of your purchase or renewal date. ...
(2 more slips)
Question: How many days do I have to ask for a refund?
That message is 438 tokens. Pasting all 24 docs instead would be 5,194 tokens, about 12× more, on every question.
9. Generate
Send the message at temperature 0. DocBot answered:
You have 15 days from your purchase or renewal date to request a full refund [refund_policy_2026.md#2].
It then added a line saying an older policy allowed 30 days. The slip ids are how we traced that line back to the stale chunk in seconds. Always ask for citations.
One more trap here: our first call used max_tokens: 300 and the answer was cut off mid-sentence. The model had spent 181 of those 300 tokens on hidden thinking. Leave room.
Build it without a framework
Here are the nine steps in about 40 lines. No LangChain, no vector database. Put your docs in a docs/ folder as markdown.
import re
from pathlib import Path
import numpy as np, tiktoken
from openai import OpenAI
from anthropic import Anthropic
openai, llm = OpenAI(), Anthropic()
enc = tiktoken.get_encoding("cl100k_base")
SYSTEM = ("Answer briefly, using only the slips below. Cite the slip id you used "
"in square brackets. If the slips don't contain the answer, say you don't know.")
def chunk(doc_name, text, limit=200):
"""Step 2: one chunk per '## ' section; long sections split at blank lines."""
out = []
for n, section in enumerate(re.split(r"\n(?=## )", text.strip())):
parts, cur = [], ""
for para in section.split("\n\n"):
cand = (cur + "\n\n" + para).strip()
if cur and len(enc.encode(cand)) > limit:
parts.append(cur); cur = para
else:
cur = cand
parts.append(cur)
out += [{"id": f"{doc_name}#{n}" + ("" if j == 0 else chr(97 + j)), "text": p}
for j, p in enumerate(parts)]
return out
def embed(texts):
"""Steps 3 and 5: same model for chunks and questions, unit length."""
r = openai.embeddings.create(model="text-embedding-3-small", input=texts)
v = np.array([d.embedding for d in r.data])
return v / np.linalg.norm(v, axis=1, keepdims=True)
# Indexing (steps 1-4): load, chunk, embed, keep in memory
chunks = [c for p in sorted(Path("docs").glob("*.md")) for c in chunk(p.name, p.read_text())]
vecs = embed([c["text"] for c in chunks])
def answer(question, k=4):
q = embed([question])[0] # 5. embed the question
top = np.argsort(-(vecs @ q))[:k] # 6. retrieve top k
slips = "\n\n".join(f"[{chunks[i]['id']}]\n{chunks[i]['text']}" for i in top)
msg = llm.messages.create( # 8 + 9. augment, generate
model="claude-haiku-5-5", max_tokens=1024, temperature=0, system=SYSTEM,
messages=[{"role": "user", "content": f"Slips:\n\n{slips}\n\nQuestion: {question}"}])
return "".join(b.text for b in msg.content if b.type == "text")
print(answer("How many days do I have to ask for a refund?"))
import { readdirSync, readFileSync } from "node:fs";
import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";
import { getEncoding } from "js-tiktoken";
const openai = new OpenAI();
const llm = new Anthropic();
const enc = getEncoding("cl100k_base");
const SYSTEM = "Answer briefly, using only the slips below. Cite the slip id you used " +
"in square brackets. If the slips don't contain the answer, say you don't know.";
// Step 2: one chunk per '## ' section; long sections split at blank lines
function chunk(docName, text, limit = 200) {
const out = [];
text.trim().split(/\n(?=## )/).forEach((section, n) => {
const parts = [];
let cur = "";
for (const para of section.split("\n\n")) {
const cand = (cur + "\n\n" + para).trim();
if (cur && enc.encode(cand).length > limit) { parts.push(cur); cur = para; }
else cur = cand;
}
parts.push(cur);
parts.forEach((p, j) =>
out.push({ id: `${docName}#${n}${j ? String.fromCharCode(97 + j) : ""}`, text: p }));
});
return out;
}
// Steps 3 and 5: same model for chunks and questions, unit length
async function embed(texts) {
const r = await openai.embeddings.create({ model: "text-embedding-3-small", input: texts });
return r.data.map(({ embedding: e }) => {
const n = Math.hypot(...e);
return e.map((x) => x / n);
});
}
const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);
// Indexing (steps 1-4): load, chunk, embed, keep in memory
const chunks = readdirSync("docs").filter((f) => f.endsWith(".md")).sort()
.flatMap((f) => chunk(f, readFileSync(`docs/${f}`, "utf8")));
const vecs = await embed(chunks.map((c) => c.text));
async function answer(question, k = 4) {
const [q] = await embed([question]); // 5. embed the question
const top = vecs.map((v, i) => ({ i, s: dot(v, q) })) // 6. retrieve top k
.sort((a, b) => b.s - a.s).slice(0, k);
const slips = top.map(({ i }) => `[${chunks[i].id}]\n${chunks[i].text}`).join("\n\n");
const msg = await llm.messages.create({ // 8 + 9. augment, generate
model: "claude-haiku-5-5", max_tokens: 1024, temperature: 0, system: SYSTEM,
messages: [{ role: "user", content: `Slips:\n\n${slips}\n\nQuestion: ${question}` }],
});
return msg.content.filter((b) => b.type === "text").map((b) => b.text).join("");
}
console.log(await answer("How many days do I have to ask for a refund?"));
Step 7 is missing on purpose: add a metadata filter once you see stale or off-limits chunks in your top 4.
Starting defaults
| Setting | DocBot’s value | Why |
|---|---|---|
| Chunking | One per ## section, split over 200 tokens |
Help docs already have one topic per section |
| Chunk id | file.md#section |
Lets the model cite, and you debug |
| Embedding model | text-embedding-3-small, 1,536 dims |
Cheap, good on English. Weak on Hinglish (see below) |
| k | 4 | Enough for one fact plus context; 438-token message |
| Temperature | 0 | Same question, same answer |
max_tokens |
1,024 | 300 got cut off by hidden thinking |
| Vector store | numpy array in memory | 67 chunks search in 4.7 µs |
These are starting points measured on one small corpus. Change one setting at a time and re-score on the same questions.
What breaks at each step
Every one of these happened on DocBot.
| Symptom | Step | What we measured | Fix |
|---|---|---|---|
| Answer mentions an old policy | 1, 7 | Stale 2025 chunk ranked #1 (0.5650) | Remove old versions, or filter on version |
| Hinglish question finds the wrong things | 3 | “paisa wapas kab milega?” vs “How long does a refund take?”: cosine 0.19 with text-embedding-3-small, 0.72 with gemini-embedding-001 |
Use a multilingual embedding model |
| Exact ID lookups fail | 3, 6 | Searching “INV-2041”: the right invoice ranked 10th of 10. INV-2041 and INV-2047 scored 0.98 alike | Add keyword search next to vectors (hybrid search) |
| Answer stops mid-sentence | 9 | 181 of 300 output tokens went to thinking | Raise max_tokens |
| Wrong answer, confident tone | 6 | The model only knows what retrieval hands it | Check the cited slip ids first |
| Chunk makes no sense alone | 2 | Chunks carry no doc title, so the year lives only in the id | Prepend the doc title to each chunk |
The last column is the order to debug in: look at the slips before you blame the model.
What a RAG pipeline costs
DocBot’s measured refund-window call: 560 input tokens and 223 output tokens (97 of them thinking), on Claude Haiku at $0.10 per million input and $0.50 per million output. That’s $0.00017 per question, or about $0.17 per 1,000 questions. Embedding the question adds about $0.0000002.
| Approach | Input tokens per question (cl100k) | Input cost per 1,000 questions |
|---|---|---|
| Paste all 24 docs | 5,194 | $0.52 |
| RAG, top 4 chunks | 438 | $0.044 |
At 24 docs, both are cheap. What scales is the ratio: double the docs and “paste everything” doubles, while RAG stays at 4 chunks.
How to know your pipeline works
Write down real questions with known answers (we use 50 for DocBot) and score two things separately:
- Retrieval: is the chunk that holds the answer in the top k? This is recall@k, and it needs no LLM to measure.
- Answer: is the answer correct, and supported by the slips it cites?
If retrieval fails, no prompt will save the answer. If retrieval passes and the answer is still wrong, then look at the prompt and the model.
When you don’t need a RAG pipeline
If all your docs fit comfortably in the context window (a few hundred pages), paste them in and turn on prompt caching so repeat questions don’t pay full price for the same text. RAG earns its place when the corpus is too big, changes often, or needs per-user permissions.
FAQ
What are the steps of a RAG pipeline?
Indexing: load, chunk, embed, store. Query: embed the question, retrieve the top k chunks, filter or rerank, build the prompt, generate. Some guides merge these into three words: retrieve, augment, generate.
Is a RAG pipeline the same as RAG architecture?
Mostly. “Pipeline” usually means the ordered steps; “architecture” adds the parts that run them: the embedding model, the vector store, the reranker, the LLM, and how indexing stays in sync with your docs.
Do I need LangChain or LlamaIndex?
No. The code above covers the whole pipeline. Frameworks help when you want loaders for many file types, ready-made rerankers, or agents. Learn the steps first, so you can tell which part of the framework is failing.
Do I need a vector database?
Not for thousands of chunks: a numpy array is fast enough. You want one when you reach hundreds of thousands of chunks, need filtering during search, or need to update vectors without rebuilding everything.
What chunk size should I use?
Start with your documents’ own structure (sections, FAQ entries). If there is none, try a few hundred tokens with a little overlap, then measure recall@k on your own questions and adjust.
Go deeper
- What is RAG?: the full DocBot story and the stale-policy bug.
- Embeddings: why the Hinglish question and the invoice IDs fail.
- Agentic RAG vs traditional RAG: what changes when the model runs the search itself.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. arXiv:2005.11401
- Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, 2016. arXiv:1603.09320
- DocBot measurements: our own runs, November 2026 (embedding
text-embedding-3-smallandgemini-embedding-001, answers from Claude Haiku, token counts withcl100k_base).