Agentic RAG vs Traditional RAG: What Changes, What It Costs
Traditional RAG searches once. Agentic RAG lets the model search as often as it needs. A real retrieval test, a 40-line agent loop, and when each one wins.
In traditional RAG, your code searches once and hands the model whatever came back. In agentic RAG, search is a tool: the model decides what to search for, reads the results, and searches again until it has enough to answer.
Same retriever, same documents. What changes is who controls the loop. This post shows the difference on a real bot, with a test we ran, the code for both, and the cost you sign up for.
The two architectures side by side
TRADITIONAL RAG (your code decides; one search)
question ──► embed ──► top 4 chunks ──► one message ──► LLM ──► answer
AGENTIC RAG (the model decides; 0 to N searches)
question ──► LLM ──┬──► "search: refund delay incident" ──► tool runs ──┐
│ │
│◄──────────────── results go back to the LLM ◄──────┘
│
├──► "search: INV-2041" ──► tool runs ──► ... (repeat)
│
└──► enough evidence ──► answer
| Traditional RAG | Agentic RAG | |
|---|---|---|
| Who writes the search query | Your code (the user’s question as-is) | The model, one query per need |
| Number of searches | Exactly 1 | 0 to N, model’s choice |
| Tools | One retriever | Any: vector search, keyword search, SQL, web, APIs |
| LLM calls per question | 1 | 2 or more |
| Latency | One round trip | One round trip per step |
| Predictability | Same steps every time | Path varies by question |
| Debugging | Read the 4 chunks | Read the whole trace |
| Fails on | Multi-part questions, exact IDs, info spread across docs | Loops, over-searching, cost spikes |
Our test: does one search miss things?
DocBot is our support bot for a made-up SaaS company: 24 help docs, 67 chunks, text-embedding-3-small, top 4 by cosine. (The full build is in the RAG pipeline post.) One doc is an incident report: on 14 March 2026 a payment-gateway problem queued 212 refunds and customers got them 1 to 2 business days late.
We asked a compound question, then split it the way an agent would.
One search for “Were refunds ever delayed, and how long does a refund normally take?”
| Chunk | Cosine |
|---|---|
refund_policy_2025.md#3 (processing time) |
0.5202 |
refund_policy_2026.md#3 (processing time) |
0.5134 |
incident_2026-03-14_payment-gateway.md#1 |
0.5041 |
incident_2026-03-14_payment-gateway.md#3 |
0.4511 |
Two searches, one per part of the question:
| Sub-query | Best hit | Cosine |
|---|---|---|
| “How long does a refund normally take?” | refund_policy_2025.md#3 |
0.5481 |
| “Were refunds delayed by an incident or outage?” | incident_2026-03-14_payment-gateway.md#0 (the summary) |
0.6438 |
The honest result: traditional RAG passed this one. The single search still pulled two incident chunks into the top 4. Splitting the question made the incident hits stronger (0.64 against 0.50) and brought in the summary chunk, but the one-shot answer would have been fine.
That’s worth saying because most posts on this topic imply classic RAG falls over on any question with an “and” in it. It doesn’t. With a small corpus and a good embedding model, one search covers a lot.
Where one search really fails
These are the failures we did measure on DocBot, and they’re the ones an agent fixes:
- Exact identifiers. We searched for invoice
INV-2041. Its own chunk ranked 10th of 10. To the embedding model, INV-2041 and INV-2047 scored 0.98 alike: vectors capture meaning, and two invoice numbers mean the same thing. An agent with a keyword search tool (or plain grep) finds it on the first try. - Answers that need a second hop. “Was the customer on INV-2041 affected by the March incident?” needs the invoice date first, then the incident window. No single query can return both, because the second query depends on the first answer.
- Facts in different systems. If the refund policy is in your docs but the order status is in a database, one vector search can’t reach the database. An agent can call both.
- Stale chunks winning. Our 2025 refund policy (30 days) outranked the 2026 one (15 days). A model that reads results can notice two conflicting versions and search for the newer one. Fixing the index is still the better fix.
If your logged failures look like these, an agent helps. If they look like “the right chunk wasn’t in the index”, it won’t.
The agentic RAG patterns, in plain words
“Agentic RAG” is an umbrella. These are the named patterns under it:
- ReAct (Yao et al., 2022): the model alternates between reasoning about what it needs and calling a tool, then reads the result and repeats. Almost every agentic RAG loop is a version of this.
- Router: one cheap LLM call picks the source (docs, SQL, web, or no retrieval at all) before anything is searched. It’s the smallest step away from classic RAG.
- Query decomposition: the model splits a compound question into sub-queries, as in our test above, and searches each.
- Self-RAG (Asai et al., 2023): the model is trained to decide when retrieval is needed and to grade whether each passage is relevant and whether its own answer is supported by it.
- Corrective RAG, CRAG (Yan et al., 2024): a light evaluator scores retrieved documents. Good ones are used, poor ones trigger a different search (in the paper, a web search), and mixed ones get both.
- Adaptive-RAG (Jeong et al., 2024): a small classifier guesses how complex the question is and routes it to no retrieval, single-step retrieval, or multi-step retrieval. Simple questions stay cheap.
Self-RAG and Adaptive-RAG need trained components. ReAct, routers and decomposition work with any model that supports tool use, which is why most production “agentic RAG” is a ReAct loop with one or two search tools.
Build the agent loop yourself
Here is agentic RAG with no framework: give the model a search_docs tool, run whatever it asks for, send the results back, stop when it answers or hits a step cap. search(query, k) is the retrieval function from the pipeline post: embed the query, return the top-k chunks with their ids.
from anthropic import Anthropic
llm = Anthropic()
MAX_STEPS = 4
TOOLS = [{
"name": "search_docs",
"description": "Search the help docs. Returns the 4 most relevant chunks with their ids. "
"Call it again with a different query if the results don't answer the question.",
"input_schema": {"type": "object",
"properties": {"query": {"type": "string"}},
"required": ["query"]},
}]
SYSTEM = ("You are DocBot. Use search_docs to find facts before answering. "
"Answer only from search results and cite chunk ids in square brackets.")
def agent_answer(question):
messages = [{"role": "user", "content": question}]
seen = set()
for step in range(MAX_STEPS):
resp = llm.messages.create(model="claude-haiku-5-5", max_tokens=1024,
system=SYSTEM, tools=TOOLS, messages=messages)
messages.append({"role": "assistant", "content": resp.content})
if resp.stop_reason != "tool_use":
return "".join(b.text for b in resp.content if b.type == "text")
results = []
for block in resp.content:
if block.type != "tool_use":
continue
q = block.input["query"].strip().lower()
if q in seen: # stop repeat searches
text = "You already ran this search. Try a different query or answer."
else:
seen.add(q)
text = "\n\n".join(f"[{c['id']}]\n{c['text']}" for c in search(q, k=4))
results.append({"type": "tool_result", "tool_use_id": block.id, "content": text})
messages.append({"role": "user", "content": results})
return "I couldn't find enough in the docs to answer that." # step cap reached
import Anthropic from "@anthropic-ai/sdk";
const llm = new Anthropic();
const MAX_STEPS = 4;
const TOOLS = [{
name: "search_docs",
description: "Search the help docs. Returns the 4 most relevant chunks with their ids. " +
"Call it again with a different query if the results don't answer the question.",
input_schema: { type: "object", properties: { query: { type: "string" } }, required: ["query"] },
}];
const SYSTEM = "You are DocBot. Use search_docs to find facts before answering. " +
"Answer only from search results and cite chunk ids in square brackets.";
async function agentAnswer(question) {
const messages = [{ role: "user", content: question }];
const seen = new Set();
for (let step = 0; step < MAX_STEPS; step++) {
const resp = await llm.messages.create({
model: "claude-haiku-5-5", max_tokens: 1024, system: SYSTEM, tools: TOOLS, messages,
});
messages.push({ role: "assistant", content: resp.content });
if (resp.stop_reason !== "tool_use") {
return resp.content.filter((b) => b.type === "text").map((b) => b.text).join("");
}
const results = [];
for (const block of resp.content) {
if (block.type !== "tool_use") continue;
const q = block.input.query.trim().toLowerCase();
let text;
if (seen.has(q)) { // stop repeat searches
text = "You already ran this search. Try a different query or answer.";
} else {
seen.add(q);
const chunks = await search(q, 4);
text = chunks.map((c) => `[${c.id}]\n${c.text}`).join("\n\n");
}
results.push({ type: "tool_result", tool_use_id: block.id, content: text });
}
messages.push({ role: "user", content: results });
}
return "I couldn't find enough in the docs to answer that."; // step cap reached
}
Compare it with the traditional version: one search() call, one messages.create(), done. The agent version is the same retriever wrapped in a loop, plus the guard rails that keep the loop from running away.
To fix the INV-2041 failure, add a second tool, keyword_search, that does an exact-match lookup. The model learns from the tool descriptions which one to use for IDs.
What the loop costs
Every turn of the loop re-sends the whole conversation so far: the question, every earlier tool call, every earlier batch of results. So input tokens grow with each step.
DocBot’s traditional RAG call measured 560 input tokens and cost about $0.17 per 1,000 questions on Claude Haiku. Here is a rough estimate for the agent version on a question that takes two searches. These are estimates, not measurements: the tool definition, the model’s own reasoning and chunk sizes all move the numbers.
| Call | What’s in the input | Input tokens (est.) |
|---|---|---|
| 1 | System + tool definition + question | ~350 |
| 2 | Call 1 + first search’s 4 chunks | ~800 |
| 3 | Call 2 + second search’s 4 chunks | ~1,250 |
| Total | ~2,400 |
That’s about 4× the input tokens of the one-shot version, three LLM round trips instead of one, and more output tokens because the model writes its search queries. On Haiku it’s still a fraction of a cent per question. On a bigger model, at a million questions a month, it’s a real line item. Prompt caching helps, since the early part of each call repeats exactly.
We’ll measure the real numbers when the RAG course races classic and agentic DocBot on the same 50 questions.
Keep the loop under control
The code above already has the first three. Production agents need all five:
- Step cap. Hard limit on loop turns (we use 4). Without it, a confused model can search forever.
- Stop condition with a fallback answer. When the cap hits, return “not found” instead of guessing.
- Repeat-query guard. Models sometimes re-run the same search. Catch it and tell the model to try something else.
- Token budget. Track input tokens across turns and stop early if a question gets expensive.
- Tracing. Log every query the model wrote and every chunk id it got back. When an answer is wrong, the trace shows whether it searched for the wrong thing or found the right thing and misread it.
Which one should you use?
Start with traditional RAG. It’s cheaper, faster, and every answer takes the same path, which makes it easy to test.
Move to agentic RAG when your logged failures cluster on:
- exact IDs, codes or names that vector search blurs together,
- questions where the second search depends on the first answer,
- facts split across different systems (docs plus a database plus an API),
- questions that sometimes need no retrieval at all.
A middle step often works: keep one search for most questions, and add a router that sends only the hard ones to the agent loop. That’s the idea behind Adaptive-RAG.
Measure both the same way
Run both versions on the same golden questions and record three numbers per question: correct or not, total tokens, and seconds to answer. Agentic RAG should win on the questions it was built for. If it doesn’t beat classic RAG on accuracy by enough to pay for the extra tokens and latency, keep the simple one.
Claude Code is agentic RAG
When Claude Code answers a question about your codebase, it doesn’t use a vector index. It runs grep and reads files, decides what to search next, and repeats. That’s agentic RAG with a keyword tool. It works well on code because code is full of exact names, which is the case where embeddings struggle most. Episode 1 of the RAG notes covers this.
FAQ
What is agentic RAG in simple words?
RAG where the model runs the search itself. It decides whether to search, what to search for, and when it has enough, instead of your code doing one fixed search before the model sees the question.
Is agentic RAG better than traditional RAG?
Better at multi-step questions, exact lookups, and pulling from several sources. Worse at cost, speed and predictability. For simple factual questions over clean docs, traditional RAG is usually the right choice.
Do I need LangGraph or another framework for agentic RAG?
No. The loop above is about 40 lines with the plain Anthropic SDK. Frameworks help when you have many tools, need retries and persistence, or want built-in tracing.
What’s the difference between agentic RAG and an AI agent?
An AI agent can use any tools to do a task. Agentic RAG is an agent whose main tools are retrieval tools, and whose job is to answer a question from your data.
How is agentic RAG different from multi-hop RAG?
Multi-hop RAG is a goal: answers that need several retrieval steps. Agentic RAG is one way to reach it, by letting the model plan the hops. You can also hard-code a fixed two-step pipeline without an agent.
Go deeper
- RAG pipeline: architecture, steps and real numbers: the one-shot pipeline this post builds on.
- What is RAG?: the DocBot story, and why Claude Code only runs grep.
- Embeddings: why INV-2041 and INV-2047 look the same to a vector.
Sources
- Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, 2022. arXiv:2210.03629
- Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, 2023. arXiv:2310.11511
- Yan et al., Corrective Retrieval Augmented Generation, 2024. arXiv:2401.15884
- Jeong et al., Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity, 2024. arXiv:2403.14403
- Singh et al., Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, 2025. arXiv:2501.09136
- Anthropic, tool use documentation.
- DocBot measurements: our own runs, November 2026 (
text-embedding-3-small, Claude Haiku).